[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:11 dist-test-slave-cq88 krb5kdc[24448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:11 dist-test-slave-cq88 krb5kdc[24448](info): set up 2 sockets
Sep 29 15:57:11 dist-test-slave-cq88 krb5kdc[24448](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 15:57:12.879189 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:57:12 dist-test-slave-cq88 krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:57:13.063597 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:35333
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:35333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:13.391716 24461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:13.392086 24461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:13.392277 24461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:13.399648 24461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:13.399822 24461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:13.399935 24461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:13.400043 24461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:13.400115 24461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:13.410723 24461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:35333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:35333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:13.412444 24461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:13.413945 24461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:13.427975 24466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:13.427927 24467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:13.427938 24469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:13.428786 24461 server_base.cc:1034] running on GCE node
I20260929 15:57:13.429679 24461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:13.431222 24461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:13.432524 24461 hybrid_clock.cc:648] HybridClock initialized: now 1790697433432480 us; error 47 us; skew 500 ppm
I20260929 15:57:13.432947 24461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:13 dist-test-slave-cq88 krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697433, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:13.436326 24461 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:57:13.437973 24461 webserver.cc:460] Webserver started at http://127.23.222.126:45887/ using document root <none> and password file <none>
I20260929 15:57:13.438710 24461 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:13.438827 24461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:13.439101 24461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:13.442308 24461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "fd28d76674114ee4aec23ed14c7aad10"
format_stamp: "Formatted at 2026-09-29 15:57:13 on dist-test-slave-cq88"
server_key: "21e286213974e19bb7a383ed4fe5bde0"
server_key_iv: "43127bbab27a223133b252d2514ce915"
server_key_version: "encryptionkey@0"
I20260929 15:57:13.443091 24461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "fd28d76674114ee4aec23ed14c7aad10"
format_stamp: "Formatted at 2026-09-29 15:57:13 on dist-test-slave-cq88"
server_key: "21e286213974e19bb7a383ed4fe5bde0"
server_key_iv: "43127bbab27a223133b252d2514ce915"
server_key_version: "encryptionkey@0"
I20260929 15:57:13.448755 24461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 15:57:13.452723 24476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:13.454273 24461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:57:13.454468 24461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "fd28d76674114ee4aec23ed14c7aad10"
format_stamp: "Formatted at 2026-09-29 15:57:13 on dist-test-slave-cq88"
server_key: "21e286213974e19bb7a383ed4fe5bde0"
server_key_iv: "43127bbab27a223133b252d2514ce915"
server_key_version: "encryptionkey@0"
I20260929 15:57:13.454658 24461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:13.470358 24461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:13.475178 24461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:13.475468 24461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:13.496759 24461 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:35333
I20260929 15:57:13.496764 24527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:35333 every 8 connection(s)
I20260929 15:57:13.498397 24461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:13.501754 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 24461
I20260929 15:57:13.502094 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:13.502624 24528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:13.504295 24441 external_mini_cluster.cc:1428] Setting key 0bc8ac0b135ecbb19d89a9c765cf97ca
I20260929 15:57:13.514674 24528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10: Bootstrap starting.
I20260929 15:57:13.517920 24528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:13.519315 24528 log.cc:824] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:13.522778 24528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10: No bootstrap required, opened a new log
I20260929 15:57:13.532068 24528 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd28d76674114ee4aec23ed14c7aad10" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 35333 } }
I20260929 15:57:13.532459 24528 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:13.532582 24528 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd28d76674114ee4aec23ed14c7aad10, State: Initialized, Role: FOLLOWER
I20260929 15:57:13.533277 24528 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd28d76674114ee4aec23ed14c7aad10" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 35333 } }
I20260929 15:57:13.533504 24528 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:13.533648 24528 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:13.533823 24528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [term 0 FOLLOWER]: Advancing to term 1
Sep 29 15:57:13 dist-test-slave-cq88 krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:13.537042 24528 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd28d76674114ee4aec23ed14c7aad10" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 35333 } }
I20260929 15:57:13.537639 24528 leader_election.cc:302] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd28d76674114ee4aec23ed14c7aad10; no voters:
I20260929 15:57:13.538408 24528 leader_election.cc:288] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:13.538838 24533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:13.541221 24528 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:13.546151 24533 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [term 1 LEADER]: Becoming Leader. State: Replica: fd28d76674114ee4aec23ed14c7aad10, State: Running, Role: LEADER
I20260929 15:57:13.546864 24533 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd28d76674114ee4aec23ed14c7aad10" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 35333 } }
I20260929 15:57:13.549817 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:13.508500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54612 (local address 127.23.222.126:35333)
0929 15:57:13.509522 (+ 1022us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:13.509545 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:13.511264 (+ 1719us) server_negotiation.cc:407] Connection header received
0929 15:57:13.520807 (+ 9543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:13.520837 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:13.521348 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:13.521830 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:13.524858 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:13.526999 (+ 2141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:13.528988 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:13.530148 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:13.530364 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:13.537128 (+ 6764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:13.537139 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:13.537155 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:13.537258 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:13.542193 (+ 4935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:13.543779 (+ 1586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:13.543793 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:13.543805 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:13.544160 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:13.545020 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:13.545028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:13.545038 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:13.546017 (+ 979us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:13.546382 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:13.547925 (+ 1543us) server_negotiation.cc:299] Negotiation successful
0929 15:57:13.548353 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":409,"threads_started":1}
I20260929 15:57:13.550376 24535 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd28d76674114ee4aec23ed14c7aad10. Latest consensus state: current_term: 1 leader_uuid: "fd28d76674114ee4aec23ed14c7aad10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd28d76674114ee4aec23ed14c7aad10" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 35333 } } }
I20260929 15:57:13.550707 24535 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:13.551254 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd28d76674114ee4aec23ed14c7aad10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd28d76674114ee4aec23ed14c7aad10" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 35333 } } }
I20260929 15:57:13.551553 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:13.552591 24549 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 15:57:13.555130 24547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:13.559422 24547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:13.568645 24547 catalog_manager.cc:1347] Generated new cluster ID: 4c24fa3652774a7ba698ec2465e37170
I20260929 15:57:13.568784 24547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:13.587802 24547 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:13.589213 24547 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:13.599649 24547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd28d76674114ee4aec23ed14c7aad10: Generated new TSK 0
I20260929 15:57:13.600633 24547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:57:13.651885 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:35333
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:13.991855 24554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:13.992194 24554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:13.992374 24554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:13.999709 24554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:13.999886 24554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:14.000083 24554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:57:14.010093 24554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:35333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:14.011510 24554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:14.012969 24554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:14.026206 24562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:14.026140 24560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:14.026229 24559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:14.027525 24554 server_base.cc:1034] running on GCE node
I20260929 15:57:14.028168 24554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:14.029273 24554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:14.030474 24554 hybrid_clock.cc:648] HybridClock initialized: now 1790697434030408 us; error 63 us; skew 500 ppm
I20260929 15:57:14.030810 24554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:14 dist-test-slave-cq88 krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697434, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:14.034315 24554 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:57:14.035933 24554 webserver.cc:460] Webserver started at http://127.23.222.65:36415/ using document root <none> and password file <none>
I20260929 15:57:14.036657 24554 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:14.036772 24554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:14.037123 24554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:14.040434 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "560108efdf6f4428a8ed629f34f637bc"
format_stamp: "Formatted at 2026-09-29 15:57:14 on dist-test-slave-cq88"
server_key: "2dc3e7d4e951ce73ef82dc59519f16f7"
server_key_iv: "07b63a62d80ae1e5caffea4d70c3a1b0"
server_key_version: "encryptionkey@0"
I20260929 15:57:14.041551 24554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "560108efdf6f4428a8ed629f34f637bc"
format_stamp: "Formatted at 2026-09-29 15:57:14 on dist-test-slave-cq88"
server_key: "2dc3e7d4e951ce73ef82dc59519f16f7"
server_key_iv: "07b63a62d80ae1e5caffea4d70c3a1b0"
server_key_version: "encryptionkey@0"
I20260929 15:57:14.047498 24554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 15:57:14.051203 24569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:14.052922 24554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:14.053133 24554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "560108efdf6f4428a8ed629f34f637bc"
format_stamp: "Formatted at 2026-09-29 15:57:14 on dist-test-slave-cq88"
server_key: "2dc3e7d4e951ce73ef82dc59519f16f7"
server_key_iv: "07b63a62d80ae1e5caffea4d70c3a1b0"
server_key_version: "encryptionkey@0"
I20260929 15:57:14.053378 24554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:14.071213 24554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:14.075872 24554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:14.076125 24554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:14.077090 24554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:14.078485 24554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:14.078562 24554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:14.078655 24554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:14.078701 24554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:14.112627 24554 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:35179
I20260929 15:57:14.112680 24681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:35179 every 8 connection(s)
I20260929 15:57:14.114082 24554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:14.120819 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 24554
I20260929 15:57:14.121065 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:14.121687 24441 external_mini_cluster.cc:1428] Setting key 07e9cdfec37be459c5a8f6737bb53cdd
Sep 29 15:57:14 dist-test-slave-cq88 krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697434, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:14.134944 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:14.116779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:46189 (local address 127.23.222.126:35333)
0929 15:57:14.117255 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:14.117260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:14.118088 (+ 828us) server_negotiation.cc:407] Connection header received
0929 15:57:14.120037 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:14.120042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:14.120131 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:14.120267 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:14.122006 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:14.122915 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:14.124578 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:14.124990 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:14.125141 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:14.128963 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:14.128968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:14.128971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:14.129004 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:14.131692 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:14.132548 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:14.132553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:14.132555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:14.132683 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:14.133145 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:14.133154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:14.133157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:14.133418 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:14.133576 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:14.134345 (+ 769us) server_negotiation.cc:299] Negotiation successful
0929 15:57:14.134571 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":256,"threads_started":1}
I20260929 15:57:14.136029 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:14.117216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:35333 (local address 127.23.222.65:46189)
0929 15:57:14.117917 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:14.117956 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:14.119749 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:14.120522 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:14.120531 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:14.121191 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:14.121830 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:14.121850 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:14.123078 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:14.123083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:14.124422 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:14.124435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:14.125243 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:14.125253 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:14.125499 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:14.126213 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:14.126237 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:14.128706 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:14.131854 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:14.131864 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:14.131933 (+ 69us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:14.132400 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:14.132803 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:14.132808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:14.132812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:14.133018 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:14.133553 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:14.133561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:14.133928 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 15:57:14.134215 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 15:57:14.134787 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":268,"threads_started":1}
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 15:57:14.137820 24682 heartbeater.cc:347] Connected to a master server at 127.23.222.126:35333
I20260929 15:57:14.138191 24682 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:14.138989 24682 heartbeater.cc:511] Master 127.23.222.126:35333 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
I20260929 15:57:14.141578 24493 ts_manager.cc:194] Registered new tserver with Master: 560108efdf6f4428a8ed629f34f637bc (127.23.222.65:35179)
I20260929 15:57:14.143374 24493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:46189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:57:14.162694 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:35333
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:14.503512 24688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:14.503829 24688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:14.503983 24688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:14.511740 24688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:14.511857 24688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:14.512042 24688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:57:14.522293 24688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:35333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:14.523794 24688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:14.525208 24688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:14.539897 24693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:14.539829 24694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:14.539868 24696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:14.540634 24688 server_base.cc:1034] running on GCE node
I20260929 15:57:14.541399 24688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:14.542641 24688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:14.543898 24688 hybrid_clock.cc:648] HybridClock initialized: now 1790697434543817 us; error 71 us; skew 500 ppm
I20260929 15:57:14.544266 24688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:14 dist-test-slave-cq88 krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697434, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:14.553637 24688 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:57:14.555397 24688 webserver.cc:460] Webserver started at http://127.23.222.66:33067/ using document root <none> and password file <none>
I20260929 15:57:14.556157 24688 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:14.556313 24688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:14.556648 24688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:14.560293 24688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "477b3c9b0d334f6a918e25953d819915"
format_stamp: "Formatted at 2026-09-29 15:57:14 on dist-test-slave-cq88"
server_key: "ea7241385ea9f7eb3f2a10b0f3d06e14"
server_key_iv: "b5f0890c5ed4b897c1230fbd7b099214"
server_key_version: "encryptionkey@0"
I20260929 15:57:14.561144 24688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "477b3c9b0d334f6a918e25953d819915"
format_stamp: "Formatted at 2026-09-29 15:57:14 on dist-test-slave-cq88"
server_key: "ea7241385ea9f7eb3f2a10b0f3d06e14"
server_key_iv: "b5f0890c5ed4b897c1230fbd7b099214"
server_key_version: "encryptionkey@0"
I20260929 15:57:14.566821 24688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 15:57:14.570923 24703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:14.572656 24688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:57:14.572880 24688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "477b3c9b0d334f6a918e25953d819915"
format_stamp: "Formatted at 2026-09-29 15:57:14 on dist-test-slave-cq88"
server_key: "ea7241385ea9f7eb3f2a10b0f3d06e14"
server_key_iv: "b5f0890c5ed4b897c1230fbd7b099214"
server_key_version: "encryptionkey@0"
I20260929 15:57:14.573105 24688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:14.588279 24688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:14.592928 24688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:14.593261 24688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:14.594375 24688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:14.595876 24688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:14.595983 24688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:14.596136 24688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:14.596218 24688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:14.632089 24688 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:43369
I20260929 15:57:14.632181 24815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:43369 every 8 connection(s)
I20260929 15:57:14.633469 24688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:14.641366 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 24688
I20260929 15:57:14.641582 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:14.642025 24441 external_mini_cluster.cc:1428] Setting key c0586b127483ddc115003a9ad9fa443e
Sep 29 15:57:14 dist-test-slave-cq88 krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697434, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:14.656754 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:14.636187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:33513 (local address 127.23.222.126:35333)
0929 15:57:14.636680 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:14.636688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:14.637874 (+ 1186us) server_negotiation.cc:407] Connection header received
0929 15:57:14.639872 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:14.639877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:14.639952 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:14.640075 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:14.642150 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:14.643120 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:14.645036 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:14.645483 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:14.645636 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:14.649520 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:14.649526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:14.649530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:14.649569 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:14.652414 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:14.653715 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:14.653720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:14.653722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:14.653867 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:14.654524 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:14.654529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:14.654531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:14.654865 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:14.654992 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:14.656169 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0929 15:57:14.656357 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":211,"threads_started":1}
I20260929 15:57:14.658473 24819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:14.636781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:35333 (local address 127.23.222.66:33513)
0929 15:57:14.637683 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:14.637729 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:14.639559 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:14.640402 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:14.640412 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:14.641034 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:14.641936 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:14.641961 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:14.643314 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:14.643320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:14.644883 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:14.644898 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:14.645740 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:14.645748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:14.645989 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:14.646720 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:14.646744 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:14.649191 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:14.652617 (+ 3426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:14.652630 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:14.652656 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:14.653532 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:14.654001 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:14.654009 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:14.654015 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:14.654347 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:14.655017 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:14.655030 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:14.655645 (+ 615us) client_negotiation.cc:769] Sending connection context
0929 15:57:14.656005 (+ 360us) client_negotiation.cc:240] Negotiation successful
0929 15:57:14.656658 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":387,"threads_started":1}
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
I20260929 15:57:14.660318 24816 heartbeater.cc:347] Connected to a master server at 127.23.222.126:35333
I20260929 15:57:14.660683 24816 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:14.661598 24816 heartbeater.cc:511] Master 127.23.222.126:35333 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
I20260929 15:57:14.663663 24493 ts_manager.cc:194] Registered new tserver with Master: 477b3c9b0d334f6a918e25953d819915 (127.23.222.66:43369)
I20260929 15:57:14.664527 24493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:33513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:57:14.687788 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:35333
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:35969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:15.019333 24822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:15.019630 24822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:15.019735 24822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:15.027098 24822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:15.027225 24822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:15.027434 24822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:57:15.037535 24822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:35333
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:15.038869 24822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:15.040231 24822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:15.054651 24827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:15.054666 24828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:15.054785 24830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:15.055495 24822 server_base.cc:1034] running on GCE node
I20260929 15:57:15.056144 24822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:15.057248 24822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:15.058501 24822 hybrid_clock.cc:648] HybridClock initialized: now 1790697435058455 us; error 56 us; skew 500 ppm
I20260929 15:57:15.058862 24822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697435, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:15.062208 24822 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:57:15.063887 24822 webserver.cc:460] Webserver started at http://127.23.222.67:46415/ using document root <none> and password file <none>
I20260929 15:57:15.064548 24822 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:15.064654 24822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:15.065026 24822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:15.068159 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "a6d0c1bd500d46c78b8f5b4fb1154f27"
format_stamp: "Formatted at 2026-09-29 15:57:15 on dist-test-slave-cq88"
server_key: "a0109c1fdd677a50a1ce34c50be1b688"
server_key_iv: "a065111e802c3f63f329cefa6a3ee283"
server_key_version: "encryptionkey@0"
I20260929 15:57:15.068958 24822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "a6d0c1bd500d46c78b8f5b4fb1154f27"
format_stamp: "Formatted at 2026-09-29 15:57:15 on dist-test-slave-cq88"
server_key: "a0109c1fdd677a50a1ce34c50be1b688"
server_key_iv: "a065111e802c3f63f329cefa6a3ee283"
server_key_version: "encryptionkey@0"
I20260929 15:57:15.074641 24822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 15:57:15.078580 24837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:15.080101 24822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:15.080279 24822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "a6d0c1bd500d46c78b8f5b4fb1154f27"
format_stamp: "Formatted at 2026-09-29 15:57:15 on dist-test-slave-cq88"
server_key: "a0109c1fdd677a50a1ce34c50be1b688"
server_key_iv: "a065111e802c3f63f329cefa6a3ee283"
server_key_version: "encryptionkey@0"
I20260929 15:57:15.080484 24822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:15.095786 24822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:15.100281 24822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:15.100601 24822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:15.101625 24822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:15.103039 24822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:15.103112 24822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:15.103299 24822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:15.103360 24822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 15:57:15.138377 24822 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:34165
I20260929 15:57:15.138437 24949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:34165 every 8 connection(s)
I20260929 15:57:15.139760 24822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:15.146536 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 24822
I20260929 15:57:15.146746 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:15.147166 24441 external_mini_cluster.cc:1428] Setting key 8a3ab635f74d507a8be41eef21cb9ca2
I20260929 15:57:15.147620 24682 heartbeater.cc:503] Master 127.23.222.126:35333 was elected leader, sending a full tablet report...
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697435, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:15.161751 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:15.142688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:44915 (local address 127.23.222.126:35333)
0929 15:57:15.142917 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:15.142922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:15.144331 (+ 1409us) server_negotiation.cc:407] Connection header received
0929 15:57:15.146271 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:15.146277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:15.146370 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:15.146482 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:15.148407 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.149609 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.151804 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.152177 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.152280 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:15.155735 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:15.155740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:15.155744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:15.155776 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:15.158593 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.159522 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.159527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.159531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.159702 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.160168 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.160172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.160174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.160490 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:15.160625 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:15.161328 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 15:57:15.161508 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:57:15.163010 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:15.143219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:35333 (local address 127.23.222.67:44915)
0929 15:57:15.144157 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:15.144199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:15.145960 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:15.146776 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:15.146785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:15.147473 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:15.148213 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:15.148247 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.149773 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.149782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:15.151674 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:15.151687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.152365 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.152369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:15.152566 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:15.153183 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:15.153212 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:15.155506 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:15.158741 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:15.158749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:15.158763 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:15.159353 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:15.159832 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:15.159835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:15.159838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:15.160024 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:15.160603 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:15.160609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:15.160953 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 15:57:15.161180 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 15:57:15.161662 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":373,"threads_started":1}
I20260929 15:57:15.164521 24950 heartbeater.cc:347] Connected to a master server at 127.23.222.126:35333
I20260929 15:57:15.164863 24950 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:15.165602 24950 heartbeater.cc:511] Master 127.23.222.126:35333 requested a full tablet report, sending...
I20260929 15:57:15.167023 24493 ts_manager.cc:194] Registered new tserver with Master: a6d0c1bd500d46c78b8f5b4fb1154f27 (127.23.222.67:34165)
I20260929 15:57:15.167917 24493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:44915
I20260929 15:57:15.177132 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697435, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697435, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:15.225473 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:15.207568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47512 (local address 127.23.222.65:35179)
0929 15:57:15.208274 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:15.208281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:15.208448 (+ 167us) server_negotiation.cc:407] Connection header received
0929 15:57:15.208673 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:15.208681 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:15.208929 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:15.209068 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:15.210597 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.211781 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.213177 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.213784 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.213902 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:15.218643 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:15.218660 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:15.218677 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:15.218725 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:15.221929 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.223019 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.223027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.223030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.223181 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.223692 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.223696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.223698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.224071 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:15.224304 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:15.224792 (+ 488us) server_negotiation.cc:299] Negotiation successful
0929 15:57:15.225078 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":377,"threads_started":1}
W20260929 15:57:15.227035 24597 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47512
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:15.274583 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:15.260026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47520 (local address 127.23.222.65:35179)
0929 15:57:15.260278 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:15.260287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:15.260629 (+ 342us) server_negotiation.cc:407] Connection header received
0929 15:57:15.260878 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:15.260882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:15.260949 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:15.261058 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:15.262428 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.263134 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.264439 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.264859 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.264990 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:15.269060 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:15.269065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:15.269069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:15.269110 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:15.271834 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.272651 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.272657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.272660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.272797 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.273184 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.273189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.273194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.273455 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:15.273718 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:15.274020 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 15:57:15.274245 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:57:15.276228 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 24554
I20260929 15:57:15.289611 24957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:15.277253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.23.222.65:35179)
0929 15:57:15.277491 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:15.277499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:15.277875 (+ 376us) server_negotiation.cc:407] Connection header received
0929 15:57:15.278059 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:15.278063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:15.278119 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:15.278219 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:15.279530 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.280231 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.281490 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.281905 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.282031 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:15.284490 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:15.284495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:15.284498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:15.284531 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:15.287182 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.287899 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.287903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.287905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.288033 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.288369 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.288372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.288374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.288619 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:15.288799 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:15.289174 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 15:57:15.289341 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 15:57:15.290361 24677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:15.377972 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 24554
I20260929 15:57:15.394651 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 24688
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:57:15.411502 24964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:15.395593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.23.222.66:43369)
0929 15:57:15.396163 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:15.396170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:15.396197 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:57:15.396257 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:15.396263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:15.396475 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:15.396606 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:15.397897 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.398926 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.400380 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.400978 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.401098 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:15.405439 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:15.405452 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:15.405469 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:15.405511 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:15.408753 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.409460 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.409467 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.409470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.409589 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.409962 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.409965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.409967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.410308 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:15.410543 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:15.410924 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 15:57:15.411134 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":287,"threads_started":1}
I20260929 15:57:15.412814 24811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:15.497093 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 24688
I20260929 15:57:15.514670 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 24822
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:57:15.530751 24966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:15.515672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.23.222.67:34165)
0929 15:57:15.516377 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:15.516384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:15.516414 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:57:15.516474 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:15.516480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:15.516704 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:15.516886 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:15.518170 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.519207 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.520542 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:15.521104 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:15.521212 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:15.525269 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:15.525280 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:15.525295 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:15.525329 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:15.528197 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.528842 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.528849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.528852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.528972 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:15.529289 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:15.529292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:15.529294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:15.529648 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:15.529884 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:15.530156 (+ 272us) server_negotiation.cc:299] Negotiation successful
0929 15:57:15.530352 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":347,"threads_started":1}
I20260929 15:57:15.532107 24945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:15.615159 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 24822
I20260929 15:57:15.631517 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 24461
I20260929 15:57:15.632680 24523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:15.736774 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 24461
2026-09-29T15:57:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3987 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24971](info): set up 2 sockets
Sep 29 15:57:15 dist-test-slave-cq88 krb5kdc[24971](info): commencing operation
krb5kdc: starting...
W20260929 15:57:16.822299 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:57:16 dist-test-slave-cq88 krb5kdc[24971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:57:17.003072 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:33163
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:17.321481 24984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:17.321784 24984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:17.321872 24984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:17.329301 24984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:17.329409 24984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:17.329455 24984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:17.329520 24984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:17.329566 24984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:17.339797 24984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:33163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:17.341118 24984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:17.342342 24984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:17.354455 24989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:17.354489 24990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:17.354537 24992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:17.355504 24984 server_base.cc:1034] running on GCE node
I20260929 15:57:17.356376 24984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:17.357901 24984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:17.359169 24984 hybrid_clock.cc:648] HybridClock initialized: now 1790697437359107 us; error 52 us; skew 500 ppm
I20260929 15:57:17.359584 24984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:17 dist-test-slave-cq88 krb5kdc[24971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:17.362768 24984 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:57:17.364383 24984 webserver.cc:460] Webserver started at http://127.23.222.126:42747/ using document root <none> and password file <none>
I20260929 15:57:17.365100 24984 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:17.365247 24984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:17.365581 24984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:17.368788 24984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "2ee9528410214823b64afdabac2fa30d"
format_stamp: "Formatted at 2026-09-29 15:57:17 on dist-test-slave-cq88"
server_key: "bc2bcb327c889c33cff881b300b89ba9"
server_key_iv: "791b6ba29aabd959974431a3aa9fc111"
server_key_version: "encryptionkey@0"
I20260929 15:57:17.369573 24984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "2ee9528410214823b64afdabac2fa30d"
format_stamp: "Formatted at 2026-09-29 15:57:17 on dist-test-slave-cq88"
server_key: "bc2bcb327c889c33cff881b300b89ba9"
server_key_iv: "791b6ba29aabd959974431a3aa9fc111"
server_key_version: "encryptionkey@0"
I20260929 15:57:17.375349 24984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 15:57:17.378811 24999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:17.380210 24984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:17.380405 24984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "2ee9528410214823b64afdabac2fa30d"
format_stamp: "Formatted at 2026-09-29 15:57:17 on dist-test-slave-cq88"
server_key: "bc2bcb327c889c33cff881b300b89ba9"
server_key_iv: "791b6ba29aabd959974431a3aa9fc111"
server_key_version: "encryptionkey@0"
I20260929 15:57:17.380617 24984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:17.402931 24984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:17.407308 24984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:17.407603 24984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:17.427085 24984 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:33163
I20260929 15:57:17.427085 25050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:33163 every 8 connection(s)
I20260929 15:57:17.428649 24984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:17.430744 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 24984
I20260929 15:57:17.430998 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:17.431874 24441 external_mini_cluster.cc:1428] Setting key 9601e11856a2b619e5d2ab992a92b183
I20260929 15:57:17.434031 25051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:17.445900 25051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d: Bootstrap starting.
Sep 29 15:57:17 dist-test-slave-cq88 krb5kdc[24971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:17.449558 25051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:17.450668 25051 log.cc:824] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:17.453758 25051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d: No bootstrap required, opened a new log
I20260929 15:57:17.455338 25054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:17.434351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50058 (local address 127.23.222.126:33163)
0929 15:57:17.435179 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:17.435192 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:17.435294 (+ 102us) server_negotiation.cc:407] Connection header received
0929 15:57:17.436986 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:17.437010 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:17.437440 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:17.437906 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:17.439502 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:17.440844 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:17.442116 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:17.442826 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:17.443045 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:17.447088 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:17.447097 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:17.447111 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:17.447154 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:17.450709 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:17.451706 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:17.451713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:17.451720 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:17.451899 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:17.452341 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:17.452345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:17.452347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:17.452914 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:17.453242 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:17.453787 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 15:57:17.454222 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":387,"threads_started":1}
I20260929 15:57:17.462231 25051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee9528410214823b64afdabac2fa30d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33163 } }
I20260929 15:57:17.462761 25051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:17.462891 25051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee9528410214823b64afdabac2fa30d, State: Initialized, Role: FOLLOWER
I20260929 15:57:17.463583 25051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee9528410214823b64afdabac2fa30d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33163 } }
I20260929 15:57:17.463830 25051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:17.463982 25051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:17.464182 25051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:17.466365 25051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee9528410214823b64afdabac2fa30d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33163 } }
I20260929 15:57:17.466889 25051 leader_election.cc:302] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ee9528410214823b64afdabac2fa30d; no voters:
I20260929 15:57:17.467543 25051 leader_election.cc:288] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:17.467682 25056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:17.468457 25056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [term 1 LEADER]: Becoming Leader. State: Replica: 2ee9528410214823b64afdabac2fa30d, State: Running, Role: LEADER
I20260929 15:57:17.468993 25056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee9528410214823b64afdabac2fa30d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33163 } }
I20260929 15:57:17.469749 25051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:17.471946 25057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ee9528410214823b64afdabac2fa30d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee9528410214823b64afdabac2fa30d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33163 } } }
I20260929 15:57:17.472434 25057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:17.473356 25063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:17.473963 25058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ee9528410214823b64afdabac2fa30d. Latest consensus state: current_term: 1 leader_uuid: "2ee9528410214823b64afdabac2fa30d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee9528410214823b64afdabac2fa30d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33163 } } }
I20260929 15:57:17.474344 25058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:17.479638 25063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:17.489238 25063 catalog_manager.cc:1347] Generated new cluster ID: fbb920217c954b29a31f3ee60b0d649c
I20260929 15:57:17.489394 25063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:17.535710 25063 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:17.537355 25063 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:17.544410 25063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ee9528410214823b64afdabac2fa30d: Generated new TSK 0
I20260929 15:57:17.545298 25063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:57:17.599851 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33163
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:17.956321 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:17.956600 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:17.956789 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:17.964082 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:17.964192 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:17.964411 25077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:57:17.974424 25077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:17.975915 25077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:17.977349 25077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:17.990392 25082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:17.990500 25085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:17.990481 25083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:17.991214 25077 server_base.cc:1034] running on GCE node
I20260929 15:57:17.991916 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:17.992856 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:17.994099 25077 hybrid_clock.cc:648] HybridClock initialized: now 1790697437994042 us; error 51 us; skew 500 ppm
I20260929 15:57:17.994472 25077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:17 dist-test-slave-cq88 krb5kdc[24971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:17.997706 25077 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:57:17.999289 25077 webserver.cc:460] Webserver started at http://127.23.222.65:34231/ using document root <none> and password file <none>
I20260929 15:57:17.999993 25077 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:18.000139 25077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:18.000459 25077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:18.003520 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "34530398d64043deb731a64bfd3d7f64"
format_stamp: "Formatted at 2026-09-29 15:57:17 on dist-test-slave-cq88"
server_key: "57cdab8cdecfdf8d57d9d0bf33b73cbb"
server_key_iv: "3ba95aada46375c6fb416c159d7e3ff3"
server_key_version: "encryptionkey@0"
I20260929 15:57:18.004366 25077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "34530398d64043deb731a64bfd3d7f64"
format_stamp: "Formatted at 2026-09-29 15:57:17 on dist-test-slave-cq88"
server_key: "57cdab8cdecfdf8d57d9d0bf33b73cbb"
server_key_iv: "3ba95aada46375c6fb416c159d7e3ff3"
server_key_version: "encryptionkey@0"
I20260929 15:57:18.009878 25077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:57:18.013321 25092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:18.014725 25077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:18.014922 25077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "34530398d64043deb731a64bfd3d7f64"
format_stamp: "Formatted at 2026-09-29 15:57:17 on dist-test-slave-cq88"
server_key: "57cdab8cdecfdf8d57d9d0bf33b73cbb"
server_key_iv: "3ba95aada46375c6fb416c159d7e3ff3"
server_key_version: "encryptionkey@0"
I20260929 15:57:18.015148 25077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:18.040071 25077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:18.044616 25077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:18.044929 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:18.045948 25077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:18.047358 25077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:18.047430 25077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:18.047574 25077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:18.047636 25077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:18.080552 25077 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:41541
I20260929 15:57:18.080596 25204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:41541 every 8 connection(s)
I20260929 15:57:18.081826 25077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:18.090119 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 25077
I20260929 15:57:18.090361 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:18.091339 24441 external_mini_cluster.cc:1428] Setting key 7de781a6f4e5f5a77df3fa95199d1691
Sep 29 15:57:18 dist-test-slave-cq88 krb5kdc[24971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697437, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:18.103135 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:18.084468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:42275 (local address 127.23.222.126:33163)
0929 15:57:18.084980 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:18.084984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:18.085978 (+ 994us) server_negotiation.cc:407] Connection header received
0929 15:57:18.087805 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:18.087809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:18.087875 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:18.087972 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:18.089398 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:18.090122 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:18.092476 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:18.092907 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:18.093094 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:18.097328 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:18.097333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:18.097336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:18.097369 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:18.100096 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:18.100839 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:18.100850 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:18.100852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:18.100967 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:18.101405 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:18.101414 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:18.101416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:18.101691 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:18.101852 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:18.102539 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 15:57:18.102752 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":235,"threads_started":1}
I20260929 15:57:18.104205 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:18.085117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:33163 (local address 127.23.222.65:42275)
0929 15:57:18.085831 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:18.085868 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:18.087556 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:18.088172 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:18.088180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:18.088659 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:18.089219 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:18.089236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:18.090361 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:18.090366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:18.092328 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:18.092342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:18.093218 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:18.093227 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:18.093490 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:18.094253 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:18.094276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:18.097090 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:18.100247 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:18.100257 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:18.100280 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:18.100706 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:18.101106 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:18.101109 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:18.101112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:18.101282 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:18.101822 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:18.101830 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:18.102158 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 15:57:18.102431 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 15:57:18.102957 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":300,"threads_started":1}
I20260929 15:57:18.105952 25205 heartbeater.cc:347] Connected to a master server at 127.23.222.126:33163
I20260929 15:57:18.106273 25205 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:18.106979 25205 heartbeater.cc:511] Master 127.23.222.126:33163 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 15:57:18.109160 25016 ts_manager.cc:194] Registered new tserver with Master: 34530398d64043deb731a64bfd3d7f64 (127.23.222.65:41541)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
I20260929 15:57:18.110845 25016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:42275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:57:18.134485 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33163
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:18.469583 25211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:18.469867 25211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:18.470046 25211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:18.477617 25211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:18.477756 25211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:18.477941 25211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:57:18.488212 25211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:18.489522 25211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:18.490818 25211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:18.503866 25216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:18.503880 25217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:18.503811 25219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:18.504850 25211 server_base.cc:1034] running on GCE node
I20260929 15:57:18.505446 25211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:18.506383 25211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:18.507591 25211 hybrid_clock.cc:648] HybridClock initialized: now 1790697438507533 us; error 40 us; skew 500 ppm
I20260929 15:57:18.507937 25211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:18 dist-test-slave-cq88 krb5kdc[24971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697438, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:18.511224 25211 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:57:18.512842 25211 webserver.cc:460] Webserver started at http://127.23.222.66:33919/ using document root <none> and password file <none>
I20260929 15:57:18.513549 25211 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:18.513689 25211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:18.514025 25211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:18.517081 25211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "37754606bb514b21a1cf99486ec3f76e"
format_stamp: "Formatted at 2026-09-29 15:57:18 on dist-test-slave-cq88"
server_key: "ba7b126418f70304ca34616b708ed59f"
server_key_iv: "e5c8c1910b1d8e6c0fdc625fc1b02f9c"
server_key_version: "encryptionkey@0"
I20260929 15:57:18.517885 25211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "37754606bb514b21a1cf99486ec3f76e"
format_stamp: "Formatted at 2026-09-29 15:57:18 on dist-test-slave-cq88"
server_key: "ba7b126418f70304ca34616b708ed59f"
server_key_iv: "e5c8c1910b1d8e6c0fdc625fc1b02f9c"
server_key_version: "encryptionkey@0"
I20260929 15:57:18.523218 25211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 15:57:18.526707 25226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:18.528039 25211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:57:18.528211 25211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "37754606bb514b21a1cf99486ec3f76e"
format_stamp: "Formatted at 2026-09-29 15:57:18 on dist-test-slave-cq88"
server_key: "ba7b126418f70304ca34616b708ed59f"
server_key_iv: "e5c8c1910b1d8e6c0fdc625fc1b02f9c"
server_key_version: "encryptionkey@0"
I20260929 15:57:18.528404 25211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:18.539685 25211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:18.543638 25211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:18.543929 25211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:18.544919 25211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:18.546204 25211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:18.546273 25211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:18.546412 25211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:18.546460 25211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:18.580819 25211 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:41853
I20260929 15:57:18.580853 25338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:41853 every 8 connection(s)
I20260929 15:57:18.582193 25211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:18.591784 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 25211
I20260929 15:57:18.591979 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:18.592429 24441 external_mini_cluster.cc:1428] Setting key 9051384e32dd292ee01e4b415aa4ffb5
Sep 29 15:57:18 dist-test-slave-cq88 krb5kdc[24971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697438, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:18.603048 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:18.585009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:37207 (local address 127.23.222.126:33163)
0929 15:57:18.585244 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:18.585249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:18.586458 (+ 1209us) server_negotiation.cc:407] Connection header received
0929 15:57:18.588476 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:18.588480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:18.588549 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:18.588638 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:18.590183 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:18.590877 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:18.592762 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:18.593380 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:18.593521 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:18.597147 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:18.597153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:18.597156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:18.597191 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:18.599907 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:18.600690 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:18.600694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:18.600697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:18.600864 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:18.601316 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:18.601327 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:18.601330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:18.601597 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:18.601753 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:18.602460 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 15:57:18.602690 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:57:18.604177 25341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:18.585538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:33163 (local address 127.23.222.66:37207)
0929 15:57:18.586287 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:18.586330 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:18.588215 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:18.588853 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:18.588861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:18.589405 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:18.590010 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:18.590039 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:18.591040 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:18.591046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:18.592603 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:18.592623 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:18.593619 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:18.593627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:18.593878 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:18.594647 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:18.594670 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:18.596900 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:18.600058 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:18.600068 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:18.600087 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:18.600517 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:18.600983 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:18.600987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:18.600991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:18.601191 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:18.601726 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:18.601734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:18.602092 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 15:57:18.602358 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 15:57:18.602921 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":291,"threads_started":1}
I20260929 15:57:18.605912 25339 heartbeater.cc:347] Connected to a master server at 127.23.222.126:33163
I20260929 15:57:18.606233 25339 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:18.606972 25339 heartbeater.cc:511] Master 127.23.222.126:33163 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
I20260929 15:57:18.608529 25016 ts_manager.cc:194] Registered new tserver with Master: 37754606bb514b21a1cf99486ec3f76e (127.23.222.66:41853)
I20260929 15:57:18.609336 25016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:37207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:57:18.635936 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33163
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:18.962913 25344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:18.963195 25344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:18.963400 25344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:18.971101 25344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:18.971275 25344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:18.971473 25344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:57:18.981710 25344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:18.983042 25344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:18.984647 25344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:18.998529 25349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:18.998554 25350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:18.998725 25352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:18.999541 25344 server_base.cc:1034] running on GCE node
I20260929 15:57:19.000272 25344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:19.001264 25344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:19.002533 25344 hybrid_clock.cc:648] HybridClock initialized: now 1790697439002463 us; error 52 us; skew 500 ppm
I20260929 15:57:19.002941 25344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:19 dist-test-slave-cq88 krb5kdc[24971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697439, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:19.006247 25344 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:57:19.007812 25344 webserver.cc:460] Webserver started at http://127.23.222.67:42753/ using document root <none> and password file <none>
I20260929 15:57:19.008471 25344 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:19.008581 25344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:19.008891 25344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:19.011963 25344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "270f5cad899f46669c3b490614c5a6dd"
format_stamp: "Formatted at 2026-09-29 15:57:19 on dist-test-slave-cq88"
server_key: "8b0fed372a39eafc2e9e74772cef46b9"
server_key_iv: "3a9235331f5d208c49d1ea6346b53862"
server_key_version: "encryptionkey@0"
I20260929 15:57:19.012723 25344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "270f5cad899f46669c3b490614c5a6dd"
format_stamp: "Formatted at 2026-09-29 15:57:19 on dist-test-slave-cq88"
server_key: "8b0fed372a39eafc2e9e74772cef46b9"
server_key_iv: "3a9235331f5d208c49d1ea6346b53862"
server_key_version: "encryptionkey@0"
I20260929 15:57:19.017988 25344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:19.022125 25359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:19.023839 25344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:57:19.024024 25344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "270f5cad899f46669c3b490614c5a6dd"
format_stamp: "Formatted at 2026-09-29 15:57:19 on dist-test-slave-cq88"
server_key: "8b0fed372a39eafc2e9e74772cef46b9"
server_key_iv: "3a9235331f5d208c49d1ea6346b53862"
server_key_version: "encryptionkey@0"
I20260929 15:57:19.024260 25344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:19.045583 25344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:19.050146 25344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:19.050483 25344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:19.051534 25344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:19.052987 25344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:19.053064 25344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:19.053246 25344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:19.053303 25344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:19.089815 25344 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:43819
I20260929 15:57:19.089886 25471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:43819 every 8 connection(s)
I20260929 15:57:19.091114 25344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:19.093544 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 25344
I20260929 15:57:19.093725 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:19.094313 24441 external_mini_cluster.cc:1428] Setting key a125c71d0013c0d604b45e5d06c56c93
Sep 29 15:57:19 dist-test-slave-cq88 krb5kdc[24971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697439, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:19.112186 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.094177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:49365 (local address 127.23.222.126:33163)
0929 15:57:19.094385 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.094391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.095659 (+ 1268us) server_negotiation.cc:407] Connection header received
0929 15:57:19.097622 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.097626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.097718 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.097809 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:19.099218 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.100014 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.101488 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.101896 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.101991 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.105938 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:19.105942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:19.105946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:19.105978 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:19.109189 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.110016 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.110020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.110023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.110165 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.110539 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.110542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.110544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.110810 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:19.110920 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.111702 (+ 782us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.111894 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:19.113046 25474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.094697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:33163 (local address 127.23.222.67:49365)
0929 15:57:19.095510 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:19.095562 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:19.097310 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:19.098011 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:19.098019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:19.098503 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:19.099062 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.099079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.100134 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.100137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.101364 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.101376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.102157 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.102164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.102391 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.103209 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:19.103285 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:19.105732 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:19.109336 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:19.109347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:19.109367 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:19.109906 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:19.110268 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:19.110272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:19.110274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:19.110445 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:19.110913 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:19.110918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:19.111295 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 15:57:19.111548 (+ 253us) client_negotiation.cc:240] Negotiation successful
0929 15:57:19.111941 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":334,"threads_started":1}
I20260929 15:57:19.113930 25205 heartbeater.cc:503] Master 127.23.222.126:33163 was elected leader, sending a full tablet report...
I20260929 15:57:19.114665 25472 heartbeater.cc:347] Connected to a master server at 127.23.222.126:33163
I20260929 15:57:19.114971 25472 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:19.115852 25472 heartbeater.cc:511] Master 127.23.222.126:33163 requested a full tablet report, sending...
I20260929 15:57:19.117149 25016 ts_manager.cc:194] Registered new tserver with Master: 270f5cad899f46669c3b490614c5a6dd (127.23.222.67:43819)
I20260929 15:57:19.117897 25016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:49365
I20260929 15:57:19.121784 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:57:19.140491 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.128072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.23.222.126:33163)
0929 15:57:19.128299 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.128303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.128864 (+ 561us) server_negotiation.cc:407] Connection header received
0929 15:57:19.129028 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.129032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.129103 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.129210 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:19.130591 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.131389 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.132673 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.133060 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.133170 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.135385 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:19.135389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:19.135393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:19.135423 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:19.137997 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.138729 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.138733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.138735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.138861 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.139199 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.139202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.139204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.139584 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:19.139769 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.140062 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.140246 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:57:19.147595 25016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:19.154531 25016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:19.185869 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.167406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:41853 (local address 127.0.0.1:54426)
0929 15:57:19.174633 (+ 7227us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:19.174690 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:19.174934 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:19.177771 (+ 2837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:19.177779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:19.177813 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:19.178323 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.178337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.180352 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.180357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.182965 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.182980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.184752 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.184758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.184946 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.185005 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 15:57:19.185155 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 15:57:19.185369 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6871,"mutex_wait_us":30,"thread_start_us":11293,"threads_started":1}
I20260929 15:57:19.186075 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.167637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54426 (local address 127.23.222.66:41853)
0929 15:57:19.177039 (+ 9402us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.177051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.177090 (+ 39us) server_negotiation.cc:407] Connection header received
0929 15:57:19.177219 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.177229 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.177513 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.177752 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:19.178541 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.180218 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.183131 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.184540 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.184705 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.184928 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.185197 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.185368 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9229,"thread_start_us":9130,"threads_started":1}
I20260929 15:57:19.187814 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.175591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44500 (local address 127.23.222.67:43819)
0929 15:57:19.178485 (+ 2894us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.178497 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.179093 (+ 596us) server_negotiation.cc:407] Connection header received
0929 15:57:19.179407 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.179414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.179636 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.179793 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:19.180705 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.182399 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.184835 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.186249 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.186355 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.186590 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.187291 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.187470 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2050,"thread_start_us":1871,"threads_started":1}
I20260929 15:57:19.187885 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.175194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:43819 (local address 127.0.0.1:44500)
0929 15:57:19.178985 (+ 3791us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:19.179000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:19.179186 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:19.179976 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:19.179980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:19.180004 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:19.180478 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.180490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.182608 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.182613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.184643 (+ 2030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.184659 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.186776 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.186784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.187070 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.187118 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 15:57:19.187201 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 15:57:19.187393 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3659,"mutex_wait_us":106,"thread_start_us":2964,"threads_started":1}
I20260929 15:57:19.188078 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.167656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:41541 (local address 127.0.0.1:56832)
0929 15:57:19.178457 (+ 10801us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:19.178476 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:19.178602 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:19.179572 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:19.179578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:19.179605 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:19.179985 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.179995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.181542 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.181549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.183815 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.183830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.186345 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.186352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.187010 (+ 658us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.187102 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 15:57:19.187191 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 15:57:19.187393 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10660,"mutex_wait_us":144,"thread_start_us":6428,"threads_started":1}
I20260929 15:57:19.189486 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.168376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56832 (local address 127.23.222.65:41541)
0929 15:57:19.177466 (+ 9090us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.177474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.178745 (+ 1271us) server_negotiation.cc:407] Connection header received
0929 15:57:19.178860 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.178867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.179104 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.179335 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:19.180166 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.181406 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.183981 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.185563 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.185696 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.185929 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.189018 (+ 3089us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.189186 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8922,"thread_start_us":9090,"threads_started":1}
I20260929 15:57:19.190320 25274 tablet_service.cc:1467] Processing CreateTablet for tablet 57259753a7874082a871f6a9f68f3ee7 (DEFAULT_TABLE table=test-table [id=3dc83fb433b644138ddfdc4101bc58da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:19.190919 25407 tablet_service.cc:1467] Processing CreateTablet for tablet 57259753a7874082a871f6a9f68f3ee7 (DEFAULT_TABLE table=test-table [id=3dc83fb433b644138ddfdc4101bc58da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:19.191726 25274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57259753a7874082a871f6a9f68f3ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:19.192481 25407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57259753a7874082a871f6a9f68f3ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:19.194516 25140 tablet_service.cc:1467] Processing CreateTablet for tablet 57259753a7874082a871f6a9f68f3ee7 (DEFAULT_TABLE table=test-table [id=3dc83fb433b644138ddfdc4101bc58da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:19.196044 25140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57259753a7874082a871f6a9f68f3ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:19.203734 25490 tablet_bootstrap.cc:492] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e: Bootstrap starting.
I20260929 15:57:19.208006 25490 tablet_bootstrap.cc:654] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:19.209079 25490 log.cc:824] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:19.211376 25491 tablet_bootstrap.cc:492] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd: Bootstrap starting.
I20260929 15:57:19.211880 25493 tablet_bootstrap.cc:492] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64: Bootstrap starting.
I20260929 15:57:19.212482 25490 tablet_bootstrap.cc:492] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e: No bootstrap required, opened a new log
I20260929 15:57:19.212770 25490 ts_tablet_manager.cc:1397] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 15:57:19.215175 25491 tablet_bootstrap.cc:654] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:19.215375 25493 tablet_bootstrap.cc:654] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:19.216284 25491 log.cc:824] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:19.216585 25493 log.cc:824] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:19.219841 25493 tablet_bootstrap.cc:492] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64: No bootstrap required, opened a new log
I20260929 15:57:19.219841 25491 tablet_bootstrap.cc:492] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd: No bootstrap required, opened a new log
I20260929 15:57:19.220125 25493 ts_tablet_manager.cc:1397] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260929 15:57:19.220126 25491 ts_tablet_manager.cc:1397] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260929 15:57:19.222008 25490 raft_consensus.cc:348] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } }
I20260929 15:57:19.222347 25490 raft_consensus.cc:374] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:19.222441 25490 raft_consensus.cc:729] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37754606bb514b21a1cf99486ec3f76e, State: Initialized, Role: FOLLOWER
I20260929 15:57:19.223172 25490 consensus_queue.cc:260] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } }
I20260929 15:57:19.225075 25339 heartbeater.cc:503] Master 127.23.222.126:33163 was elected leader, sending a full tablet report...
I20260929 15:57:19.225519 25490 ts_tablet_manager.cc:1428] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260929 15:57:19.229842 25491 raft_consensus.cc:348] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } }
I20260929 15:57:19.230290 25491 raft_consensus.cc:374] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:19.230401 25491 raft_consensus.cc:729] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270f5cad899f46669c3b490614c5a6dd, State: Initialized, Role: FOLLOWER
I20260929 15:57:19.231014 25491 consensus_queue.cc:260] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } }
I20260929 15:57:19.231276 25493 raft_consensus.cc:348] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } }
I20260929 15:57:19.231777 25493 raft_consensus.cc:374] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:19.231952 25493 raft_consensus.cc:729] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34530398d64043deb731a64bfd3d7f64, State: Initialized, Role: FOLLOWER
I20260929 15:57:19.232712 25472 heartbeater.cc:503] Master 127.23.222.126:33163 was elected leader, sending a full tablet report...
I20260929 15:57:19.232573 25493 consensus_queue.cc:260] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } }
I20260929 15:57:19.233831 25491 ts_tablet_manager.cc:1428] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 15:57:19.234303 25493 ts_tablet_manager.cc:1428] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260929 15:57:19.278822 25498 raft_consensus.cc:482] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:19.279145 25498 raft_consensus.cc:504] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } }
I20260929 15:57:19.280954 25498 leader_election.cc:288] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37754606bb514b21a1cf99486ec3f76e (127.23.222.66:41853), 270f5cad899f46669c3b490614c5a6dd (127.23.222.67:43819)
I20260929 15:57:19.289004 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.281611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:46691 (local address 127.23.222.67:43819)
0929 15:57:19.281841 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.281848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.282417 (+ 569us) server_negotiation.cc:407] Connection header received
0929 15:57:19.282613 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.282618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.282691 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.282771 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:19.283611 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.285030 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.287038 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.288189 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.288356 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.288436 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.288625 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.288752 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 15:57:19.289057 25499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.281259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:43819 (local address 127.23.222.65:46691)
0929 15:57:19.282298 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:19.282313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:19.282464 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:19.282897 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:19.282905 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:19.282930 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:19.283404 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.283421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.285156 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.285162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.286866 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.286879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.288368 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.288372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.288479 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.288502 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 15:57:19.288575 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 15:57:19.288661 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":101,"thread_start_us":483,"threads_started":1}
I20260929 15:57:19.290010 25500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.281200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:41853 (local address 127.23.222.65:54579)
0929 15:57:19.282184 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:19.282205 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:19.282401 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:19.282801 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:19.282805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:19.283097 (+ 292us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:19.283667 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.283682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.285121 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.285128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.287356 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:19.287368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.288965 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.288971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:19.289566 (+ 595us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.289591 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 15:57:19.289662 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 15:57:19.289753 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":30,"spinlock_wait_cycles":1280,"thread_start_us":829,"threads_started":1}
I20260929 15:57:19.290125 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.281295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:54579 (local address 127.23.222.66:41853)
0929 15:57:19.281540 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.281550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.282319 (+ 769us) server_negotiation.cc:407] Connection header received
0929 15:57:19.282607 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.282611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.282674 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.282779 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:19.283826 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.284940 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.287508 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.288744 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.288898 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.288961 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.289729 (+ 768us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.289866 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:57:19.290092 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57259753a7874082a871f6a9f68f3ee7" candidate_uuid: "34530398d64043deb731a64bfd3d7f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270f5cad899f46669c3b490614c5a6dd" is_pre_election: true
I20260929 15:57:19.290725 25427 raft_consensus.cc:2454] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34530398d64043deb731a64bfd3d7f64 in term 0.
I20260929 15:57:19.290962 25294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57259753a7874082a871f6a9f68f3ee7" candidate_uuid: "34530398d64043deb731a64bfd3d7f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37754606bb514b21a1cf99486ec3f76e" is_pre_election: true
I20260929 15:57:19.291424 25096 leader_election.cc:302] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 270f5cad899f46669c3b490614c5a6dd, 34530398d64043deb731a64bfd3d7f64; no voters:
I20260929 15:57:19.291501 25294 raft_consensus.cc:2454] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34530398d64043deb731a64bfd3d7f64 in term 0.
I20260929 15:57:19.291837 25498 raft_consensus.cc:2781] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:19.291998 25498 raft_consensus.cc:482] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:19.292122 25498 raft_consensus.cc:3037] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:19.294293 25498 raft_consensus.cc:504] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } }
I20260929 15:57:19.294896 25498 leader_election.cc:288] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [CANDIDATE]: Term 1 election: Requested vote from peers 37754606bb514b21a1cf99486ec3f76e (127.23.222.66:41853), 270f5cad899f46669c3b490614c5a6dd (127.23.222.67:43819)
I20260929 15:57:19.295284 25294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57259753a7874082a871f6a9f68f3ee7" candidate_uuid: "34530398d64043deb731a64bfd3d7f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37754606bb514b21a1cf99486ec3f76e"
I20260929 15:57:19.295429 25427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57259753a7874082a871f6a9f68f3ee7" candidate_uuid: "34530398d64043deb731a64bfd3d7f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270f5cad899f46669c3b490614c5a6dd"
I20260929 15:57:19.295495 25294 raft_consensus.cc:3037] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:19.295629 25427 raft_consensus.cc:3037] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:19.298748 25427 raft_consensus.cc:2454] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34530398d64043deb731a64bfd3d7f64 in term 1.
I20260929 15:57:19.298748 25294 raft_consensus.cc:2454] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34530398d64043deb731a64bfd3d7f64 in term 1.
I20260929 15:57:19.299149 25093 leader_election.cc:302] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34530398d64043deb731a64bfd3d7f64, 37754606bb514b21a1cf99486ec3f76e; no voters:
I20260929 15:57:19.299606 25498 raft_consensus.cc:2781] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:19.300051 25498 raft_consensus.cc:686] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [term 1 LEADER]: Becoming Leader. State: Replica: 34530398d64043deb731a64bfd3d7f64, State: Running, Role: LEADER
I20260929 15:57:19.300511 25498 consensus_queue.cc:237] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } }
I20260929 15:57:19.304997 25015 catalog_manager.cc:5495] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34530398d64043deb731a64bfd3d7f64 (127.23.222.65). New cstate: current_term: 1 leader_uuid: "34530398d64043deb731a64bfd3d7f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37754606bb514b21a1cf99486ec3f76e" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 41853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34530398d64043deb731a64bfd3d7f64" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "270f5cad899f46669c3b490614c5a6dd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43819 } health_report { overall_health: UNKNOWN } } }
W20260929 15:57:19.334373 25206 tablet.cc:2369] T 57259753a7874082a871f6a9f68f3ee7 P 34530398d64043deb731a64bfd3d7f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:57:19.334450 25340 tablet.cc:2369] T 57259753a7874082a871f6a9f68f3ee7 P 37754606bb514b21a1cf99486ec3f76e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:57:19.343518 25473 tablet.cc:2369] T 57259753a7874082a871f6a9f68f3ee7 P 270f5cad899f46669c3b490614c5a6dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 15:57:19 dist-test-slave-cq88 krb5kdc[24971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:19.360502 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.340426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.23.222.65:41541)
0929 15:57:19.340639 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.340647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.341062 (+ 415us) server_negotiation.cc:407] Connection header received
0929 15:57:19.341262 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.341275 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.341350 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.341457 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:19.343073 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.343975 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.345269 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.345695 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.345832 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.354413 (+ 8581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:19.354431 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:19.354448 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:19.354498 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:19.357453 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.358267 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.358275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.358277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.358453 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.358863 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.358868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.358873 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.359283 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:19.359609 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.359892 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.360157 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 15:57:19 dist-test-slave-cq88 krb5kdc[24971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:57:19 dist-test-slave-cq88 krb5kdc[24971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697439, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:19.405856 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.392352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.23.222.65:41541)
0929 15:57:19.392589 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.392597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.392956 (+ 359us) server_negotiation.cc:407] Connection header received
0929 15:57:19.393113 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.393117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.393192 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.393284 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:19.394614 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.395427 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.396655 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.397065 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.397160 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.400927 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:19.400932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:19.400935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:19.400973 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:19.403613 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.404193 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.404197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.404199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.404323 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.404676 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.404679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.404681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.404931 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:19.405146 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.405333 (+ 187us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.405494 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 15:57:19 dist-test-slave-cq88 krb5kdc[24971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:57:19 dist-test-slave-cq88 krb5kdc[24971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:19.460139 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.442414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.23.222.65:41541)
0929 15:57:19.442611 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.442620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.443021 (+ 401us) server_negotiation.cc:407] Connection header received
0929 15:57:19.443284 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.443290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.443387 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.443504 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:19.444854 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.445602 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.447332 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.448052 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.448192 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.453177 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:19.453185 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:19.453190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:19.453242 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:19.457032 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.457723 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.457731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.457735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.457915 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.458418 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.458424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.458430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.458870 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:19.459146 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.459542 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.459774 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 15:57:19.462406 25120 tablet_service.cc:2908] Rejecting scan request for tablet 57259753a7874082a871f6a9f68f3ee7: Uninitialized: clean time has not yet been initialized
I20260929 15:57:19.466441 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25077
I20260929 15:57:19.482409 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.467322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.23.222.65:41541)
0929 15:57:19.467522 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.467530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.467872 (+ 342us) server_negotiation.cc:407] Connection header received
0929 15:57:19.468091 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.468095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.468163 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.468240 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:19.470012 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.471291 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.473373 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.474026 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.474169 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.477165 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:19.477175 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:19.477178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:19.477211 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:19.479839 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.480501 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.480507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.480511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.480651 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.481018 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.481023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.481027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.481274 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:19.481527 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.481817 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.482059 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 15:57:19.483104 25200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:19.574946 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 25077
I20260929 15:57:19.593140 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25211
Sep 29 15:57:19 dist-test-slave-cq88 krb5kdc[24971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:57:19.616269 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.594069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.23.222.66:41853)
0929 15:57:19.594249 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.594253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.594271 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:57:19.594366 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.594392 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.594470 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.594578 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:19.595796 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.596589 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.597813 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.598244 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.598342 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.610641 (+ 12299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:19.610659 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:19.610677 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:19.610722 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:19.613679 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.614437 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.614444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.614446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.614573 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.614910 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.614913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.614915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.615356 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:19.615592 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.615841 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.616008 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:57:19.617024 25334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:19.707085 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 25211
I20260929 15:57:19.724149 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25344
Sep 29 15:57:19 dist-test-slave-cq88 krb5kdc[24971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:57:19.739220 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:19.725021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44502 (local address 127.23.222.67:43819)
0929 15:57:19.725215 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:19.725220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:19.725242 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:57:19.725528 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:19.725533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:19.725618 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:19.725714 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:19.726935 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.727788 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.729072 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:19.729576 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:19.729722 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:19.733944 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:19.733959 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:19.733975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:19.734013 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:19.736782 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.737383 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.737391 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.737393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.737542 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:19.737903 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:19.737906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:19.737908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:19.738266 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:19.738477 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:19.738767 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 15:57:19.738961 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:57:19.739910 25467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:19.827664 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 25344
I20260929 15:57:19.844452 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 24984
I20260929 15:57:19.845651 25046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:19.955420 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 24984
2026-09-29T15:57:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4219 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:20 dist-test-slave-cq88 krb5kdc[25520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:20 dist-test-slave-cq88 krb5kdc[25520](info): set up 2 sockets
Sep 29 15:57:20 dist-test-slave-cq88 krb5kdc[25520](info): commencing operation
krb5kdc: starting...
W20260929 15:57:21.046182 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:57:21 dist-test-slave-cq88 krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:57:21.231945 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34615
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:34615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:21.553319 25533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:21.553651 25533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:21.553814 25533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:21.560992 25533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:21.561136 25533 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:57:21.561234 25533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:21.561353 25533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:21.561439 25533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:21.561537 25533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:21.571408 25533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:34615
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:21.572737 25533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:21.574061 25533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:21.586175 25538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:21.586144 25541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:21.586146 25539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:21.587149 25533 server_base.cc:1034] running on GCE node
I20260929 15:57:21.587999 25533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:21.589517 25533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:21.590759 25533 hybrid_clock.cc:648] HybridClock initialized: now 1790697441590699 us; error 36 us; skew 500 ppm
I20260929 15:57:21.591048 25533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:21 dist-test-slave-cq88 krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:21.594226 25533 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:57:21.595711 25533 webserver.cc:460] Webserver started at http://127.23.222.126:41869/ using document root <none> and password file <none>
I20260929 15:57:21.596395 25533 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:21.596478 25533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:21.596755 25533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:21.599713 25533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "ac8e63f6a15d4096aea8fd6127cd778e"
format_stamp: "Formatted at 2026-09-29 15:57:21 on dist-test-slave-cq88"
server_key: "36a42726470bced5c59b5201c58de619"
server_key_iv: "3c9a12e1f0f624de245346538642bbff"
server_key_version: "encryptionkey@0"
I20260929 15:57:21.600423 25533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "ac8e63f6a15d4096aea8fd6127cd778e"
format_stamp: "Formatted at 2026-09-29 15:57:21 on dist-test-slave-cq88"
server_key: "36a42726470bced5c59b5201c58de619"
server_key_iv: "3c9a12e1f0f624de245346538642bbff"
server_key_version: "encryptionkey@0"
I20260929 15:57:21.605480 25533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 15:57:21.608861 25548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:21.610373 25533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 15:57:21.610514 25533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "ac8e63f6a15d4096aea8fd6127cd778e"
format_stamp: "Formatted at 2026-09-29 15:57:21 on dist-test-slave-cq88"
server_key: "36a42726470bced5c59b5201c58de619"
server_key_iv: "3c9a12e1f0f624de245346538642bbff"
server_key_version: "encryptionkey@0"
I20260929 15:57:21.610697 25533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:21.636224 25533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:21.640825 25533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:21.641047 25533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:21.662673 25533 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:34615
I20260929 15:57:21.662716 25609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:34615 every 8 connection(s)
I20260929 15:57:21.664120 25533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:21.667991 25610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:21.669919 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 25533
I20260929 15:57:21.670156 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:21.670797 24441 external_mini_cluster.cc:1428] Setting key 1c8e0d0c6d21e4ffefb1782befa7cc33
I20260929 15:57:21.681283 25610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e: Bootstrap starting.
I20260929 15:57:21.684749 25610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:21.685727 25610 log.cc:824] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:57:21 dist-test-slave-cq88 krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:21.689191 25610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e: No bootstrap required, opened a new log
I20260929 15:57:21.695896 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:21.673019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.23.222.126:34615)
0929 15:57:21.673869 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:21.673886 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:21.673938 (+ 52us) server_negotiation.cc:407] Connection header received
0929 15:57:21.676307 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:21.676338 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:21.676851 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:21.677291 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:21.678623 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:21.680584 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:21.681849 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:21.682842 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:21.683074 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:21.687321 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:21.687331 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:21.687350 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:21.687400 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:21.690810 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:21.691656 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:21.691675 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:21.691682 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:21.691858 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:21.692248 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:21.692253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:21.692256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:21.692804 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:21.693155 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:21.693587 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 15:57:21.694298 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":350,"threads_started":1}
I20260929 15:57:21.698355 25610 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e63f6a15d4096aea8fd6127cd778e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34615 } }
I20260929 15:57:21.698769 25610 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:21.698895 25610 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8e63f6a15d4096aea8fd6127cd778e, State: Initialized, Role: FOLLOWER
I20260929 15:57:21.699596 25610 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e63f6a15d4096aea8fd6127cd778e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34615 } }
I20260929 15:57:21.699843 25610 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:21.699993 25610 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:21.700179 25610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:21.702684 25610 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e63f6a15d4096aea8fd6127cd778e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34615 } }
I20260929 15:57:21.703315 25610 leader_election.cc:302] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac8e63f6a15d4096aea8fd6127cd778e; no voters:
I20260929 15:57:21.703904 25610 leader_election.cc:288] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:21.704062 25615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:21.704864 25615 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [term 1 LEADER]: Becoming Leader. State: Replica: ac8e63f6a15d4096aea8fd6127cd778e, State: Running, Role: LEADER
I20260929 15:57:21.705386 25615 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e63f6a15d4096aea8fd6127cd778e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34615 } }
I20260929 15:57:21.706216 25610 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:21.708052 25617 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac8e63f6a15d4096aea8fd6127cd778e. Latest consensus state: current_term: 1 leader_uuid: "ac8e63f6a15d4096aea8fd6127cd778e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e63f6a15d4096aea8fd6127cd778e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34615 } } }
I20260929 15:57:21.708425 25617 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:21.708840 25616 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac8e63f6a15d4096aea8fd6127cd778e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8e63f6a15d4096aea8fd6127cd778e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34615 } } }
I20260929 15:57:21.709213 25616 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:21.711105 25620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:21.716847 25620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:21.726346 25620 catalog_manager.cc:1347] Generated new cluster ID: 0b6c3f643952443488be20afa7f586db
I20260929 15:57:21.726470 25620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:21.763226 25620 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:21.765007 25620 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:21.779348 25620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e: Generated new TSK 0
I20260929 15:57:21.780601 25620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:57:21.830792 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34615
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:22.156307 25636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:22.156577 25636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:22.156673 25636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:22.163827 25636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:22.163981 25636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:22.164134 25636 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:22.164268 25636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:57:22.173892 25636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.222.126:34615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:22.175290 25636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:22.176612 25636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:22.192843 25642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:22.192844 25644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:22.193506 25641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:22.193684 25636 server_base.cc:1034] running on GCE node
I20260929 15:57:22.194352 25636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:22.195384 25636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:22.196606 25636 hybrid_clock.cc:648] HybridClock initialized: now 1790697442196539 us; error 46 us; skew 500 ppm
I20260929 15:57:22.196911 25636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:22 dist-test-slave-cq88 krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:22.200115 25636 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:57:22.201638 25636 webserver.cc:460] Webserver started at http://127.23.222.65:41037/ using document root <none> and password file <none>
I20260929 15:57:22.202355 25636 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:22.202482 25636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:22.202813 25636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:22.205946 25636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "9f9d95238185425da7120f39dd62a64f"
format_stamp: "Formatted at 2026-09-29 15:57:22 on dist-test-slave-cq88"
server_key: "8cbdb745c89cd1e1cb1513e8a61ade38"
server_key_iv: "aeb0c52a8fd4aadd216087d7a23790ab"
server_key_version: "encryptionkey@0"
I20260929 15:57:22.206756 25636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "9f9d95238185425da7120f39dd62a64f"
format_stamp: "Formatted at 2026-09-29 15:57:22 on dist-test-slave-cq88"
server_key: "8cbdb745c89cd1e1cb1513e8a61ade38"
server_key_iv: "aeb0c52a8fd4aadd216087d7a23790ab"
server_key_version: "encryptionkey@0"
I20260929 15:57:22.212280 25636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 15:57:22.216126 25651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:22.217458 25636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 15:57:22.217671 25636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "9f9d95238185425da7120f39dd62a64f"
format_stamp: "Formatted at 2026-09-29 15:57:22 on dist-test-slave-cq88"
server_key: "8cbdb745c89cd1e1cb1513e8a61ade38"
server_key_iv: "aeb0c52a8fd4aadd216087d7a23790ab"
server_key_version: "encryptionkey@0"
I20260929 15:57:22.217911 25636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:22.237011 25636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:22.241489 25636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:22.241704 25636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:22.244364 25636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:22.244452 25636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:22.244625 25636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:22.244684 25636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:22 dist-test-slave-cq88 krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:22.269941 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:22.244963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:50857 (local address 127.23.222.126:34615)
0929 15:57:22.245552 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:22.245559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:22.246747 (+ 1188us) server_negotiation.cc:407] Connection header received
0929 15:57:22.249836 (+ 3089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:22.249842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:22.249940 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:22.250093 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:22.252899 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.253907 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.257176 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.257827 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.257982 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:22.262884 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:22.262893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:22.262901 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:22.262966 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:22.266434 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:22.267453 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:22.267459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:22.267462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:22.267629 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:22.268166 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:22.268171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:22.268175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:22.268457 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:22.268627 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:22.269340 (+ 713us) server_negotiation.cc:299] Negotiation successful
0929 15:57:22.269568 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":345,"threads_started":1}
I20260929 15:57:22.270977 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:22.245651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34615 (local address 127.23.222.65:50857)
0929 15:57:22.246549 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:22.246601 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:22.249455 (+ 2854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:22.250416 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:22.250434 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:22.251566 (+ 1132us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:22.252671 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:22.252705 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.254108 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.254115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:22.256999 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:22.257020 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.258138 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.258147 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:22.258538 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:22.259628 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:22.259665 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:22.262606 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:22.266584 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:22.266600 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:22.266621 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:22.267300 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:22.267775 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:22.267782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:22.267786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:22.268033 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:22.268609 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:22.268618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:22.268980 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 15:57:22.269246 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 15:57:22.269779 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":364,"threads_started":1}
I20260929 15:57:22.291569 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:22.276921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34615 (local address 127.23.222.65:49713)
0929 15:57:22.277564 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:22.277579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:22.277737 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:22.278263 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:22.278267 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:22.278768 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:22.279106 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:22.279115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.280251 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.280258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:22.282022 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:22.282043 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.283275 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.283288 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:22.283451 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:22.284152 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:22.284168 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:22.284889 (+ 721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:22.288941 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:22.288945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:22.288963 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:22.289357 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:22.289744 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:22.289747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:22.289749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:22.289857 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:22.290422 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:22.290425 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:22.290594 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 15:57:22.290703 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:57:22.291134 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":450,"threads_started":1}
I20260929 15:57:22.291611 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:22.277322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:49713 (local address 127.23.222.126:34615)
0929 15:57:22.277518 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:22.277524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:22.277654 (+ 130us) server_negotiation.cc:407] Connection header received
0929 15:57:22.277868 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:22.277871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:22.277976 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:22.278100 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:22.279281 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.280027 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.282202 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.283013 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.283172 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:22.285099 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:22.285109 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:22.285117 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:22.285165 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:22.288827 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:22.289470 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:22.289479 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:22.289485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:22.289638 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:22.289971 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:22.289975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:22.289978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:22.290314 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:22.290481 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:22.290860 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 15:57:22.291112 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:57:22.301986 25636 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:41679
I20260929 15:57:22.302031 25771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:41679 every 8 connection(s)
I20260929 15:57:22.303181 25636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:22.305197 25772 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34615
I20260929 15:57:22.305428 25772 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:22.306177 25772 heartbeater.cc:511] Master 127.23.222.126:34615 requested a full tablet report, sending...
I20260929 15:57:22.308308 25565 ts_manager.cc:194] Registered new tserver with Master: 9f9d95238185425da7120f39dd62a64f (127.23.222.65:41679)
I20260929 15:57:22.309660 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 25636
I20260929 15:57:22.309882 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:22.309850 25565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:50857
I20260929 15:57:22.310551 24441 external_mini_cluster.cc:1428] Setting key a6979d6fe2b6fbcbe13f39c28c30f412
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:57:22.351879 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34615
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:22.674700 25776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:22.675005 25776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:22.675180 25776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:22.682515 25776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:22.682663 25776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:22.682821 25776 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:22.682961 25776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:57:22.692981 25776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.222.126:34615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:22.694321 25776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:22.695694 25776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:22.710548 25784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:22.710626 25781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:22.710623 25782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:22.711642 25776 server_base.cc:1034] running on GCE node
I20260929 15:57:22.712321 25776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:22.713275 25776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:22.714494 25776 hybrid_clock.cc:648] HybridClock initialized: now 1790697442714436 us; error 35 us; skew 500 ppm
I20260929 15:57:22.714860 25776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:22 dist-test-slave-cq88 krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:22.718199 25776 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:57:22.719693 25776 webserver.cc:460] Webserver started at http://127.23.222.66:43119/ using document root <none> and password file <none>
I20260929 15:57:22.720400 25776 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:22.720520 25776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:22.720849 25776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:22.724041 25776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "e30e5427bd0e463fa29d776eb7dc3890"
format_stamp: "Formatted at 2026-09-29 15:57:22 on dist-test-slave-cq88"
server_key: "0b0d2f03344d691cd2e7b800186ab924"
server_key_iv: "e394ecb0a5666fe07497de85bd0c010c"
server_key_version: "encryptionkey@0"
I20260929 15:57:22.724871 25776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "e30e5427bd0e463fa29d776eb7dc3890"
format_stamp: "Formatted at 2026-09-29 15:57:22 on dist-test-slave-cq88"
server_key: "0b0d2f03344d691cd2e7b800186ab924"
server_key_iv: "e394ecb0a5666fe07497de85bd0c010c"
server_key_version: "encryptionkey@0"
I20260929 15:57:22.730285 25776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:22.734167 25791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:22.735756 25776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:22.735934 25776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "e30e5427bd0e463fa29d776eb7dc3890"
format_stamp: "Formatted at 2026-09-29 15:57:22 on dist-test-slave-cq88"
server_key: "0b0d2f03344d691cd2e7b800186ab924"
server_key_iv: "e394ecb0a5666fe07497de85bd0c010c"
server_key_version: "encryptionkey@0"
I20260929 15:57:22.736176 25776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:22.748118 25776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:22.752391 25776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:22.752692 25776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:22.755533 25776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:22.755615 25776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:22.755766 25776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:22.755837 25776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:22 dist-test-slave-cq88 krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:22.783651 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:22.757477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34615 (local address 127.23.222.66:45823)
0929 15:57:22.758379 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:22.758420 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:22.760492 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:22.761366 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:22.761379 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:22.762254 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:22.763296 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:22.763330 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.764701 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.764708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:22.767139 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:22.767158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.768316 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.768330 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:22.768705 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:22.769588 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:22.769612 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:22.772816 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:22.777527 (+ 4711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:22.777540 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:22.777556 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:22.778037 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:22.778606 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:22.778610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:22.778612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:22.778853 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:22.779677 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:22.779692 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:22.780411 (+ 719us) client_negotiation.cc:769] Sending connection context
0929 15:57:22.780811 (+ 400us) client_negotiation.cc:240] Negotiation successful
0929 15:57:22.781555 (+ 744us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":396,"threads_started":1}
I20260929 15:57:22.785498 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:22.756454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:45823 (local address 127.23.222.126:34615)
0929 15:57:22.756664 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:22.756672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:22.758585 (+ 1913us) server_negotiation.cc:407] Connection header received
0929 15:57:22.760831 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:22.760837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:22.760938 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:22.761056 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:22.763529 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.764526 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.767387 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.767853 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.768196 (+ 343us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:22.773149 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:22.773155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:22.773158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:22.773194 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:22.777318 (+ 4124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:22.778199 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:22.778208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:22.778215 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:22.778417 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:22.778981 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:22.778989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:22.778994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:22.779502 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:22.784542 (+ 5040us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:22.784732 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 15:57:22.785059 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:57:22.808531 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:22.791304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:49965 (local address 127.23.222.126:34615)
0929 15:57:22.791525 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:22.791533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:22.792399 (+ 866us) server_negotiation.cc:407] Connection header received
0929 15:57:22.792606 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:22.792615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:22.792713 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:22.792815 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:22.794341 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.795413 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.798060 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.798518 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.798630 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:22.801540 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:22.801550 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:22.801556 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:22.801625 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:22.805069 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:22.806092 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:22.806097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:22.806099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:22.806301 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:22.806818 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:22.806828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:22.806830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:22.807211 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:22.807431 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:22.808032 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 15:57:22.808269 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:22.809123 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:22.791460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34615 (local address 127.23.222.66:49965)
0929 15:57:22.792278 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:22.792293 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:22.792463 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:22.792899 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:22.792907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:22.793649 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:22.794137 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:22.794156 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.795627 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.795636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:22.797855 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:22.797876 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:22.798752 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:22.798762 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:22.799039 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:22.799889 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:22.799911 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:22.801279 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:22.805226 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:22.805235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:22.805242 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:22.805944 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:22.806431 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:22.806438 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:22.806444 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:22.806641 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:22.807399 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:22.807407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:22.807681 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 15:57:22.807896 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 15:57:22.808380 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":558,"threads_started":1}
I20260929 15:57:22.814397 25776 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:35113
I20260929 15:57:22.814437 25910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:35113 every 8 connection(s)
I20260929 15:57:22.815773 25776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:22.817979 25911 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34615
I20260929 15:57:22.818251 25911 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:22.818941 25911 heartbeater.cc:511] Master 127.23.222.126:34615 requested a full tablet report, sending...
I20260929 15:57:22.820386 25565 ts_manager.cc:194] Registered new tserver with Master: e30e5427bd0e463fa29d776eb7dc3890 (127.23.222.66:35113)
I20260929 15:57:22.821318 25565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:45823
I20260929 15:57:22.821863 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 25776
I20260929 15:57:22.822062 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:22.822520 24441 external_mini_cluster.cc:1428] Setting key 212705291e674336f8cd922a3240930e
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:57:22.863556 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34615
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:23.192210 25915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:23.192488 25915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:23.192682 25915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:23.200134 25915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:23.200243 25915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:23.200412 25915 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:23.200543 25915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:57:23.210853 25915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.222.126:34615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:23.212371 25915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:23.213815 25915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:23.227140 25923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:23.227200 25920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:23.227226 25921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:23.228374 25915 server_base.cc:1034] running on GCE node
I20260929 15:57:23.229007 25915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:23.229931 25915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:23.231179 25915 hybrid_clock.cc:648] HybridClock initialized: now 1790697443231095 us; error 59 us; skew 500 ppm
I20260929 15:57:23.231504 25915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:23 dist-test-slave-cq88 krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:23.234769 25915 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:57:23.236238 25915 webserver.cc:460] Webserver started at http://127.23.222.67:46859/ using document root <none> and password file <none>
I20260929 15:57:23.236965 25915 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:23.237098 25915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:23.237417 25915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:23.240355 25915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "a4945197a85248daa2e963d97512307c"
format_stamp: "Formatted at 2026-09-29 15:57:23 on dist-test-slave-cq88"
server_key: "055fcee86cc41cd239e63a761787c316"
server_key_iv: "fcc4fa3d7de58a19a95d96bb64db8a32"
server_key_version: "encryptionkey@0"
I20260929 15:57:23.241190 25915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "a4945197a85248daa2e963d97512307c"
format_stamp: "Formatted at 2026-09-29 15:57:23 on dist-test-slave-cq88"
server_key: "055fcee86cc41cd239e63a761787c316"
server_key_iv: "fcc4fa3d7de58a19a95d96bb64db8a32"
server_key_version: "encryptionkey@0"
I20260929 15:57:23.246248 25915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:57:23.249662 25930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:23.250950 25915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:23.251122 25915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "a4945197a85248daa2e963d97512307c"
format_stamp: "Formatted at 2026-09-29 15:57:23 on dist-test-slave-cq88"
server_key: "055fcee86cc41cd239e63a761787c316"
server_key_iv: "fcc4fa3d7de58a19a95d96bb64db8a32"
server_key_version: "encryptionkey@0"
I20260929 15:57:23.251376 25915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:23.274119 25915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:23.278286 25915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:23.278522 25915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:23.281122 25915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:23.281244 25915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:23.281402 25915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:23.281481 25915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:23 dist-test-slave-cq88 krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:23.307389 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.281995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:47937 (local address 127.23.222.126:34615)
0929 15:57:23.282184 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.282190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.283703 (+ 1513us) server_negotiation.cc:407] Connection header received
0929 15:57:23.287085 (+ 3382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.287092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.287194 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.287443 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:23.289927 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.291304 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.293038 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.293789 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.293912 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.298945 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:23.298952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:23.298959 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:23.299010 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:23.303150 (+ 4140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.304104 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.304112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.304119 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.304335 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.304879 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.304887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.304893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.305318 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:23.305495 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.306414 (+ 919us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.306760 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:57:23.308426 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.282469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34615 (local address 127.23.222.67:47937)
0929 15:57:23.283489 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.283551 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.286543 (+ 2992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.287727 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.287736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.288711 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:23.289663 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.289688 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.291552 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.291555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.292877 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.292889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.294018 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.294023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.294241 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.295052 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:23.295100 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:23.298708 (+ 3608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:23.303346 (+ 4638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:23.303356 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:23.303373 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:23.303964 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:23.304495 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:23.304500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:23.304505 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:23.304751 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:23.305485 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:23.305493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:23.305930 (+ 437us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.306239 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.306786 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":393,"threads_started":1}
I20260929 15:57:23.313319 25772 heartbeater.cc:503] Master 127.23.222.126:34615 was elected leader, sending a full tablet report...
I20260929 15:57:23.329175 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.315556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:43061 (local address 127.23.222.126:34615)
0929 15:57:23.315783 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.315791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.315814 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:57:23.315880 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.315887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.316001 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.316140 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:23.317181 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.318628 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.320284 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.320898 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.321067 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.322898 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:23.322906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:23.322914 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:23.322943 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:23.326423 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.327131 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.327140 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.327144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.327359 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.327839 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.327843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.327845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.328113 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:23.328233 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.328788 (+ 555us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.328947 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 15:57:23.329761 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.314874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34615 (local address 127.23.222.67:43061)
0929 15:57:23.315587 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.315613 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.315710 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.316183 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.316186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.316719 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:23.317028 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.317037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.318783 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.318787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.320135 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.320146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.321115 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.321121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.321314 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.321940 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:23.322004 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:23.322705 (+ 701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:23.326577 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:23.326581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:23.326586 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:23.327002 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:23.327491 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:23.327501 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:23.327505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:23.327706 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:23.328232 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:23.328240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:23.328501 (+ 261us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.328693 (+ 192us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.329155 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":396,"threads_started":1}
I20260929 15:57:23.347127 25915 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:36655
I20260929 15:57:23.347155 26049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:36655 every 8 connection(s)
I20260929 15:57:23.348482 25915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:23.350656 26050 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34615
I20260929 15:57:23.350889 26050 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:23.351616 26050 heartbeater.cc:511] Master 127.23.222.126:34615 requested a full tablet report, sending...
I20260929 15:57:23.353024 25565 ts_manager.cc:194] Registered new tserver with Master: a4945197a85248daa2e963d97512307c (127.23.222.67:36655)
I20260929 15:57:23.353857 25565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:47937
I20260929 15:57:23.358577 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 25915
I20260929 15:57:23.358740 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:23.359138 24441 external_mini_cluster.cc:1428] Setting key 2f75e4c246ee36f813cc105c3dade93c
I20260929 15:57:23.362129 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:23 dist-test-slave-cq88 krb5kdc[25520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:57:23 dist-test-slave-cq88 krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:23.408677 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.393845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.23.222.126:34615)
0929 15:57:23.394122 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.394131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.394455 (+ 324us) server_negotiation.cc:407] Connection header received
0929 15:57:23.394686 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.394690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.394775 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.394890 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:23.396472 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.397273 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.398665 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.399094 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.399191 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.403210 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:23.403215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:23.403219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:23.403298 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:23.405819 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.406562 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.406566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.406569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.406708 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.407072 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.407075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.407077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.407697 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:23.407887 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.408196 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.408374 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 15:57:23.413147 25565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:23.420239 25565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:23.459098 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.432773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35113 (local address 127.0.0.1:52240)
0929 15:57:23.450532 (+ 17759us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.450565 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.450942 (+ 377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.451787 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.451802 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.451837 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:23.452437 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.452453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.454184 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.454189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.456672 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.456684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.458046 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.458052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.458265 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.458320 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.458463 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.458686 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17522,"mutex_wait_us":83,"thread_start_us":18963,"threads_started":1}
I20260929 15:57:23.459889 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.433104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.23.222.66:35113)
0929 15:57:23.450542 (+ 17438us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.450555 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.450825 (+ 270us) server_negotiation.cc:407] Connection header received
0929 15:57:23.451206 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.451212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.451499 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.452673 (+ 1174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:23.452888 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.454043 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.456825 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.457826 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.457977 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.458216 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.458537 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.458723 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17263,"thread_start_us":17144,"threads_started":1}
I20260929 15:57:23.460254 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.432662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:36655 (local address 127.0.0.1:42484)
0929 15:57:23.449327 (+ 16665us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.449382 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.449596 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.451987 (+ 2391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.451993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.452030 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:23.452608 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.452621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.455019 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.455024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.457523 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.457540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.459388 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.459395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.459577 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.459615 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.459684 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.459814 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16393,"mutex_wait_us":35,"spinlock_wait_cycles":15232,"thread_start_us":16214,"threads_started":1}
I20260929 15:57:23.460870 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.435614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.23.222.65:41679)
0929 15:57:23.441517 (+ 5903us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.441531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.452200 (+ 10669us) server_negotiation.cc:407] Connection header received
0929 15:57:23.452553 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.452563 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.452806 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.452975 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:23.453847 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.455012 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.457726 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.459164 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.459637 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.459841 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.460436 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.460607 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5696,"thread_start_us":5551,"threads_started":1}
I20260929 15:57:23.461206 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.433130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.23.222.67:36655)
0929 15:57:23.451200 (+ 18070us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.451212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.451286 (+ 74us) server_negotiation.cc:407] Connection header received
0929 15:57:23.451410 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.451419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.451751 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.452805 (+ 1054us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:23.453143 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.454860 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.457706 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.459122 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.460551 (+ 1429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.460715 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.460767 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.460877 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17850,"thread_start_us":17651,"threads_started":1}
I20260929 15:57:23.461620 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.434869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:41679 (local address 127.0.0.1:58860)
0929 15:57:23.452052 (+ 17183us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.452081 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.452312 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.453233 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.453238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.453267 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:23.453625 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.453640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.455193 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.455199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.457552 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.457561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.460028 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.460037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.460220 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.460259 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.461149 (+ 890us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.461287 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17043,"mutex_wait_us":106,"thread_start_us":15147,"threads_started":1}
I20260929 15:57:23.464341 25841 tablet_service.cc:1467] Processing CreateTablet for tablet 492ea657efe04a7fab3eea0a318dc9e8 (DEFAULT_TABLE table=test-table [id=3ab8b4fa16604d88ba289f9dca467785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:23.464784 25980 tablet_service.cc:1467] Processing CreateTablet for tablet 492ea657efe04a7fab3eea0a318dc9e8 (DEFAULT_TABLE table=test-table [id=3ab8b4fa16604d88ba289f9dca467785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:23.465257 25702 tablet_service.cc:1467] Processing CreateTablet for tablet 492ea657efe04a7fab3eea0a318dc9e8 (DEFAULT_TABLE table=test-table [id=3ab8b4fa16604d88ba289f9dca467785]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:23.465842 25841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492ea657efe04a7fab3eea0a318dc9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.466042 25980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492ea657efe04a7fab3eea0a318dc9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.466496 25702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 492ea657efe04a7fab3eea0a318dc9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.477244 26069 tablet_bootstrap.cc:492] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: Bootstrap starting.
I20260929 15:57:23.480288 26070 tablet_bootstrap.cc:492] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: Bootstrap starting.
I20260929 15:57:23.481312 26071 tablet_bootstrap.cc:492] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: Bootstrap starting.
I20260929 15:57:23.482492 26069 tablet_bootstrap.cc:654] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.483749 26070 tablet_bootstrap.cc:654] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.483820 26069 log.cc:824] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:23.484066 26071 tablet_bootstrap.cc:654] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.484941 26070 log.cc:824] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:23.485287 26071 log.cc:824] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:23.487955 26069 tablet_bootstrap.cc:492] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: No bootstrap required, opened a new log
I20260929 15:57:23.488233 26069 ts_tablet_manager.cc:1397] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 15:57:23.489524 26071 tablet_bootstrap.cc:492] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: No bootstrap required, opened a new log
I20260929 15:57:23.489780 26071 ts_tablet_manager.cc:1397] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 15:57:23.489847 26070 tablet_bootstrap.cc:492] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: No bootstrap required, opened a new log
I20260929 15:57:23.490135 26070 ts_tablet_manager.cc:1397] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 15:57:23.497386 26071 raft_consensus.cc:348] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } }
I20260929 15:57:23.497766 26071 raft_consensus.cc:374] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.497889 26071 raft_consensus.cc:729] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30e5427bd0e463fa29d776eb7dc3890, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.498546 26071 consensus_queue.cc:260] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } }
I20260929 15:57:23.498725 26069 raft_consensus.cc:348] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } }
I20260929 15:57:23.499140 26069 raft_consensus.cc:374] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.499229 26069 raft_consensus.cc:729] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4945197a85248daa2e963d97512307c, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.499830 25911 heartbeater.cc:503] Master 127.23.222.126:34615 was elected leader, sending a full tablet report...
I20260929 15:57:23.500305 26071 ts_tablet_manager.cc:1428] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260929 15:57:23.500465 26070 raft_consensus.cc:348] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } }
I20260929 15:57:23.500857 26070 raft_consensus.cc:374] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.500768 26069 consensus_queue.cc:260] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } }
I20260929 15:57:23.500949 26070 raft_consensus.cc:729] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9d95238185425da7120f39dd62a64f, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.501667 26070 consensus_queue.cc:260] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } }
I20260929 15:57:23.502525 26069 ts_tablet_manager.cc:1428] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260929 15:57:23.503326 26070 ts_tablet_manager.cc:1428] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20260929 15:57:23.503746 26050 heartbeater.cc:503] Master 127.23.222.126:34615 was elected leader, sending a full tablet report...
W20260929 15:57:23.555788 25773 tablet.cc:2369] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:23.563124 26077 raft_consensus.cc:482] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:23.563484 26077 raft_consensus.cc:504] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } }
I20260929 15:57:23.565196 26077 leader_election.cc:288] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4945197a85248daa2e963d97512307c (127.23.222.67:36655), e30e5427bd0e463fa29d776eb7dc3890 (127.23.222.66:35113)
W20260929 15:57:23.567855 25912 tablet.cc:2369] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:23.574476 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.565801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:56271 (local address 127.23.222.66:35113)
0929 15:57:23.566016 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.566024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.566686 (+ 662us) server_negotiation.cc:407] Connection header received
0929 15:57:23.566886 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.566891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.566968 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.567073 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:23.567891 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.568937 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.571965 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.573232 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.573440 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.573516 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.574056 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.574207 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:23.574474 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.565607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:57431 (local address 127.23.222.67:36655)
0929 15:57:23.565835 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.565841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.566754 (+ 913us) server_negotiation.cc:407] Connection header received
0929 15:57:23.566885 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.566890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.566968 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.567073 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:23.567996 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.568758 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.571964 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.573290 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.573440 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.573545 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.574057 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.574207 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:57:23.574801 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.565551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35113 (local address 127.23.222.65:56271)
0929 15:57:23.566560 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.566581 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.566747 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.567141 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.567147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.567179 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:23.567696 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.567713 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.569061 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.569066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.571774 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.571790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.573528 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.573539 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.573823 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.573912 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.574016 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.574244 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"mutex_wait_us":303,"thread_start_us":523,"threads_started":1}
I20260929 15:57:23.575201 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.565710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:36655 (local address 127.23.222.65:57431)
0929 15:57:23.566651 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.566664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.566759 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.567141 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.567147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.567271 (+ 124us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:23.567767 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.567780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.568900 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.568905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.571774 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.571790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.573583 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.573589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.573822 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.573909 (+ 87us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.574016 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.574244 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":279,"spinlock_wait_cycles":22784,"thread_start_us":564,"threads_started":1}
I20260929 15:57:23.575959 25861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492ea657efe04a7fab3eea0a318dc9e8" candidate_uuid: "9f9d95238185425da7120f39dd62a64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30e5427bd0e463fa29d776eb7dc3890" is_pre_election: true
I20260929 15:57:23.576118 26005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492ea657efe04a7fab3eea0a318dc9e8" candidate_uuid: "9f9d95238185425da7120f39dd62a64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4945197a85248daa2e963d97512307c" is_pre_election: true
I20260929 15:57:23.576514 25861 raft_consensus.cc:2454] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9d95238185425da7120f39dd62a64f in term 0.
I20260929 15:57:23.576648 26005 raft_consensus.cc:2454] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9d95238185425da7120f39dd62a64f in term 0.
I20260929 15:57:23.577245 25655 leader_election.cc:302] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9d95238185425da7120f39dd62a64f, e30e5427bd0e463fa29d776eb7dc3890; no voters:
I20260929 15:57:23.577657 26077 raft_consensus.cc:2781] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:23.577803 26077 raft_consensus.cc:482] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:23.577906 26077 raft_consensus.cc:3037] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:23.580415 26077 raft_consensus.cc:504] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } }
I20260929 15:57:23.581020 26077 leader_election.cc:288] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [CANDIDATE]: Term 1 election: Requested vote from peers a4945197a85248daa2e963d97512307c (127.23.222.67:36655), e30e5427bd0e463fa29d776eb7dc3890 (127.23.222.66:35113)
I20260929 15:57:23.581462 26005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492ea657efe04a7fab3eea0a318dc9e8" candidate_uuid: "9f9d95238185425da7120f39dd62a64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4945197a85248daa2e963d97512307c"
I20260929 15:57:23.581487 25861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "492ea657efe04a7fab3eea0a318dc9e8" candidate_uuid: "9f9d95238185425da7120f39dd62a64f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30e5427bd0e463fa29d776eb7dc3890"
I20260929 15:57:23.581672 26005 raft_consensus.cc:3037] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:23.581688 25861 raft_consensus.cc:3037] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:23.583963 26005 raft_consensus.cc:2454] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9d95238185425da7120f39dd62a64f in term 1.
I20260929 15:57:23.584003 25861 raft_consensus.cc:2454] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9d95238185425da7120f39dd62a64f in term 1.
I20260929 15:57:23.584441 25654 leader_election.cc:302] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9d95238185425da7120f39dd62a64f, a4945197a85248daa2e963d97512307c; no voters:
I20260929 15:57:23.584825 26077 raft_consensus.cc:2781] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:23.585292 26077 raft_consensus.cc:686] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 1 LEADER]: Becoming Leader. State: Replica: 9f9d95238185425da7120f39dd62a64f, State: Running, Role: LEADER
I20260929 15:57:23.585850 26077 consensus_queue.cc:237] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } }
I20260929 15:57:23.590638 25565 catalog_manager.cc:5495] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f9d95238185425da7120f39dd62a64f (127.23.222.65). New cstate: current_term: 1 leader_uuid: "9f9d95238185425da7120f39dd62a64f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } health_report { overall_health: HEALTHY } } }
W20260929 15:57:23.600068 26051 tablet.cc:2369] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:23.628867 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.622990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.23.222.65:41679)
0929 15:57:23.623185 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.623189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.623380 (+ 191us) server_negotiation.cc:407] Connection header received
0929 15:57:23.623744 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.623748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.623855 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.624014 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:23.624428 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.625177 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.626551 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.626971 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.627077 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.627761 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:23.628323 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:23.628436 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.628591 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.628661 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:57:23.637136 26005 raft_consensus.cc:1261] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [term 1 FOLLOWER]: Refusing update from remote peer 9f9d95238185425da7120f39dd62a64f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:23.637152 25861 raft_consensus.cc:1261] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Refusing update from remote peer 9f9d95238185425da7120f39dd62a64f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:23.638146 26077 consensus_queue.cc:1035] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:23.638463 26080 consensus_queue.cc:1035] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:23.651435 26083 mvcc.cc:205] Tried to move back new op lower bound from 7334696729127682048 to 7334696728931549184. Current Snapshot: MvccSnapshot[applied={T|T < 7334696729127682048}]
I20260929 15:57:23.669957 25565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:23.670269 25565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:23.674865 25565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e: Sending DeleteTablet for 3 replicas of tablet 492ea657efe04a7fab3eea0a318dc9e8
I20260929 15:57:23.676721 25980 tablet_service.cc:1514] Processing DeleteTablet for tablet 492ea657efe04a7fab3eea0a318dc9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:23 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:42484
I20260929 15:57:23.677700 25702 tablet_service.cc:1514] Processing DeleteTablet for tablet 492ea657efe04a7fab3eea0a318dc9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:23 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:58860
I20260929 15:57:23.678934 25841 tablet_service.cc:1514] Processing DeleteTablet for tablet 492ea657efe04a7fab3eea0a318dc9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:23 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:52240
I20260929 15:57:23.683671 25565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:23.684563 25565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:23.687494 26093 tablet_replica.cc:330] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: stopping tablet replica
I20260929 15:57:23.687932 26093 raft_consensus.cc:2229] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:23.688453 26093 raft_consensus.cc:2258] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:23.689248 26095 tablet_replica.cc:330] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: stopping tablet replica
I20260929 15:57:23.689668 26095 raft_consensus.cc:2229] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:23.690632 26094 tablet_replica.cc:330] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: stopping tablet replica
I20260929 15:57:23.691037 26094 raft_consensus.cc:2229] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:23.691797 26094 raft_consensus.cc:2258] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:23.692243 26095 raft_consensus.cc:2258] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:23.692476 26093 ts_tablet_manager.cc:1905] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:23.694360 26095 ts_tablet_manager.cc:1905] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:23.696549 26094 ts_tablet_manager.cc:1905] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:23.697619 25980 tablet_service.cc:1467] Processing CreateTablet for tablet ee9751312ac24d4ba9e51d79e757559f (DEFAULT_TABLE table=test-table [id=72b3a8eb560948569035f9a5344a581d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:23.698062 25980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee9751312ac24d4ba9e51d79e757559f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.698765 25702 tablet_service.cc:1467] Processing CreateTablet for tablet ee9751312ac24d4ba9e51d79e757559f (DEFAULT_TABLE table=test-table [id=72b3a8eb560948569035f9a5344a581d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:23.699484 25841 tablet_service.cc:1467] Processing CreateTablet for tablet ee9751312ac24d4ba9e51d79e757559f (DEFAULT_TABLE table=test-table [id=72b3a8eb560948569035f9a5344a581d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:23.700009 25702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee9751312ac24d4ba9e51d79e757559f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.700382 25841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee9751312ac24d4ba9e51d79e757559f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.706416 26069 tablet_bootstrap.cc:492] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: Bootstrap starting.
I20260929 15:57:23.708164 26069 tablet_bootstrap.cc:654] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.713634 26094 ts_tablet_manager.cc:1918] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:23.713856 26094 log.cc:1196] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/492ea657efe04a7fab3eea0a318dc9e8
I20260929 15:57:23.714529 26094 ts_tablet_manager.cc:1939] T 492ea657efe04a7fab3eea0a318dc9e8 P 9f9d95238185425da7120f39dd62a64f: Deleting consensus metadata
I20260929 15:57:23.714887 26095 ts_tablet_manager.cc:1918] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:23.715072 26095 log.cc:1196] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/492ea657efe04a7fab3eea0a318dc9e8
I20260929 15:57:23.715623 26095 ts_tablet_manager.cc:1939] T 492ea657efe04a7fab3eea0a318dc9e8 P e30e5427bd0e463fa29d776eb7dc3890: Deleting consensus metadata
I20260929 15:57:23.716828 26093 ts_tablet_manager.cc:1918] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:23.717012 26093 log.cc:1196] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/492ea657efe04a7fab3eea0a318dc9e8
I20260929 15:57:23.717576 26093 ts_tablet_manager.cc:1939] T 492ea657efe04a7fab3eea0a318dc9e8 P a4945197a85248daa2e963d97512307c: Deleting consensus metadata
I20260929 15:57:23.717658 25552 catalog_manager.cc:4841] TS e30e5427bd0e463fa29d776eb7dc3890 (127.23.222.66:35113): tablet 492ea657efe04a7fab3eea0a318dc9e8 (table test-table [id=3ab8b4fa16604d88ba289f9dca467785]) successfully deleted
I20260929 15:57:23.719359 25551 catalog_manager.cc:4841] TS a4945197a85248daa2e963d97512307c (127.23.222.67:36655): tablet 492ea657efe04a7fab3eea0a318dc9e8 (table test-table [id=3ab8b4fa16604d88ba289f9dca467785]) successfully deleted
I20260929 15:57:23.719599 25550 catalog_manager.cc:4841] TS 9f9d95238185425da7120f39dd62a64f (127.23.222.65:41679): tablet 492ea657efe04a7fab3eea0a318dc9e8 (table test-table [id=3ab8b4fa16604d88ba289f9dca467785]) successfully deleted
I20260929 15:57:23.722783 26071 tablet_bootstrap.cc:492] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: Bootstrap starting.
I20260929 15:57:23.722875 26070 tablet_bootstrap.cc:492] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: Bootstrap starting.
I20260929 15:57:23.723062 26069 tablet_bootstrap.cc:492] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: No bootstrap required, opened a new log
I20260929 15:57:23.723206 26069 ts_tablet_manager.cc:1397] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260929 15:57:23.724323 26069 raft_consensus.cc:348] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.724582 26070 tablet_bootstrap.cc:654] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.724575 26071 tablet_bootstrap.cc:654] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.724575 26069 raft_consensus.cc:374] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.724663 26069 raft_consensus.cc:729] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4945197a85248daa2e963d97512307c, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.724998 26069 consensus_queue.cc:260] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.728022 26069 ts_tablet_manager.cc:1428] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 15:57:23.736954 26071 tablet_bootstrap.cc:492] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: No bootstrap required, opened a new log
I20260929 15:57:23.736964 26070 tablet_bootstrap.cc:492] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: No bootstrap required, opened a new log
I20260929 15:57:23.737154 26071 ts_tablet_manager.cc:1397] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260929 15:57:23.737159 26070 ts_tablet_manager.cc:1397] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260929 15:57:23.738010 26070 raft_consensus.cc:348] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.737989 26071 raft_consensus.cc:348] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.738231 26071 raft_consensus.cc:374] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.738238 26070 raft_consensus.cc:374] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.738332 26071 raft_consensus.cc:729] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30e5427bd0e463fa29d776eb7dc3890, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.738337 26070 raft_consensus.cc:729] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9d95238185425da7120f39dd62a64f, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.738613 26070 consensus_queue.cc:260] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.738612 26071 consensus_queue.cc:260] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.739789 26071 ts_tablet_manager.cc:1428] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:23.739789 26070 ts_tablet_manager.cc:1428] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 15:57:23.746340 26076 raft_consensus.cc:482] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:23.746582 26076 raft_consensus.cc:504] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.748220 26076 leader_election.cc:288] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f9d95238185425da7120f39dd62a64f (127.23.222.65:41679), e30e5427bd0e463fa29d776eb7dc3890 (127.23.222.66:35113)
I20260929 15:57:23.756559 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.748473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:41679 (local address 127.23.222.67:59821)
0929 15:57:23.748716 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.748737 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.748888 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.749342 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.749347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.749385 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:23.749678 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.749694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.751209 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.751217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.753979 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.754003 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.755832 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.755840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.756048 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.756094 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.756192 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.756348 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 15:57:23.757189 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.748865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:59821 (local address 127.23.222.65:41679)
0929 15:57:23.749027 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.749033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.749057 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:57:23.749129 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.749134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.749203 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.749299 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:23.749846 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.751066 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.754265 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.755656 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.755855 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.755932 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.756813 (+ 881us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.756949 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 15:57:23.758136 25726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee9751312ac24d4ba9e51d79e757559f" candidate_uuid: "a4945197a85248daa2e963d97512307c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9d95238185425da7120f39dd62a64f" is_pre_election: true
I20260929 15:57:23.758663 25726 raft_consensus.cc:2454] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4945197a85248daa2e963d97512307c in term 0.
I20260929 15:57:23.759429 25932 leader_election.cc:302] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9d95238185425da7120f39dd62a64f, a4945197a85248daa2e963d97512307c; no voters:
I20260929 15:57:23.759934 26076 raft_consensus.cc:2781] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:23.760075 26076 raft_consensus.cc:482] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:23.760190 26076 raft_consensus.cc:3037] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:23.762261 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.749120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:44763 (local address 127.23.222.66:35113)
0929 15:57:23.749312 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.749317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.754735 (+ 5418us) server_negotiation.cc:407] Connection header received
0929 15:57:23.754935 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.754941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.755027 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.755126 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:23.755923 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.757101 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.759863 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.761156 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.761331 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.761413 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.761830 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.761979 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:57:23.762311 26099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.748714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35113 (local address 127.23.222.67:44763)
0929 15:57:23.754601 (+ 5887us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.754623 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.754797 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.755159 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.755164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.755263 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:23.755692 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.755704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.757505 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.757509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.759605 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.759625 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.761414 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.761422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.761628 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.761666 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.761763 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.761924 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5717,"thread_start_us":5565,"threads_started":1}
I20260929 15:57:23.763023 26076 raft_consensus.cc:504] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.763722 26076 leader_election.cc:288] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [CANDIDATE]: Term 1 election: Requested vote from peers 9f9d95238185425da7120f39dd62a64f (127.23.222.65:41679), e30e5427bd0e463fa29d776eb7dc3890 (127.23.222.66:35113)
I20260929 15:57:23.764233 25726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee9751312ac24d4ba9e51d79e757559f" candidate_uuid: "a4945197a85248daa2e963d97512307c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9d95238185425da7120f39dd62a64f"
I20260929 15:57:23.764488 25726 raft_consensus.cc:3037] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:23.764529 25861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee9751312ac24d4ba9e51d79e757559f" candidate_uuid: "a4945197a85248daa2e963d97512307c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30e5427bd0e463fa29d776eb7dc3890" is_pre_election: true
I20260929 15:57:23.764791 25861 raft_consensus.cc:2454] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4945197a85248daa2e963d97512307c in term 0.
I20260929 15:57:23.764683 25860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee9751312ac24d4ba9e51d79e757559f" candidate_uuid: "a4945197a85248daa2e963d97512307c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30e5427bd0e463fa29d776eb7dc3890"
I20260929 15:57:23.764987 25860 raft_consensus.cc:3037] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:23.767335 25726 raft_consensus.cc:2454] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4945197a85248daa2e963d97512307c in term 1.
I20260929 15:57:23.767819 25932 leader_election.cc:302] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9d95238185425da7120f39dd62a64f, a4945197a85248daa2e963d97512307c; no voters:
I20260929 15:57:23.768129 26076 raft_consensus.cc:2781] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:23.768591 25860 raft_consensus.cc:2454] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4945197a85248daa2e963d97512307c in term 1.
I20260929 15:57:23.768870 26076 raft_consensus.cc:686] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 1 LEADER]: Becoming Leader. State: Replica: a4945197a85248daa2e963d97512307c, State: Running, Role: LEADER
I20260929 15:57:23.769294 26076 consensus_queue.cc:237] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.773164 25565 catalog_manager.cc:5495] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c reported cstate change: term changed from 0 to 1, leader changed from <none> to a4945197a85248daa2e963d97512307c (127.23.222.67). New cstate: current_term: 1 leader_uuid: "a4945197a85248daa2e963d97512307c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } health_report { overall_health: HEALTHY } } }
Sep 29 15:57:23 dist-test-slave-cq88 krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:23.831595 26107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.814110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34615 (local address 127.0.0.1:53920)
0929 15:57:23.814907 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.814927 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.815079 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.815539 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.815545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.816441 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:23.816783 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.816795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.818256 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.818262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.819829 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.819843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.820786 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.820792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.820998 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.821800 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:23.821826 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:23.824846 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:23.828530 (+ 3684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:23.828543 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:23.828551 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:23.829116 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:23.829564 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:23.829569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:23.829573 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:23.829756 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:23.830384 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:23.830392 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:23.830602 (+ 210us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.830784 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.831084 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":454,"threads_started":1}
I20260929 15:57:23.831632 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.814373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.23.222.126:34615)
0929 15:57:23.814980 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.814986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.815009 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:57:23.815221 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.815227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.815328 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.815438 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:23.816957 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.818114 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.819982 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.820597 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.820726 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.825049 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:23.825056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:23.825061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:23.825094 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:23.828376 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.829247 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.829252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.829255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.829414 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:23.829871 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:23.829876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:23.829880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:23.830210 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:23.830354 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.830891 (+ 537us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.831281 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":486}
I20260929 15:57:23.837486 25565 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:53920:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:57:23.838693 25565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:23.845497 25841 tablet_service.cc:1467] Processing CreateTablet for tablet b8f2a931119c41b08a6cd10db41ec922 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56bf172cf6054537ac7613a8d8d76d3f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:23.845475 25702 tablet_service.cc:1467] Processing CreateTablet for tablet b8f2a931119c41b08a6cd10db41ec922 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56bf172cf6054537ac7613a8d8d76d3f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:23.846076 25702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f2a931119c41b08a6cd10db41ec922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.846091 25841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f2a931119c41b08a6cd10db41ec922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.845870 25980 tablet_service.cc:1467] Processing CreateTablet for tablet b8f2a931119c41b08a6cd10db41ec922 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=56bf172cf6054537ac7613a8d8d76d3f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:23.846441 25980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f2a931119c41b08a6cd10db41ec922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:23.853415 26070 tablet_bootstrap.cc:492] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: Bootstrap starting.
I20260929 15:57:23.853505 26071 tablet_bootstrap.cc:492] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: Bootstrap starting.
I20260929 15:57:23.855259 26071 tablet_bootstrap.cc:654] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.855283 26070 tablet_bootstrap.cc:654] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.855521 26069 tablet_bootstrap.cc:492] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: Bootstrap starting.
I20260929 15:57:23.857179 26069 tablet_bootstrap.cc:654] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:23.865593 26069 tablet_bootstrap.cc:492] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: No bootstrap required, opened a new log
I20260929 15:57:23.865818 26069 ts_tablet_manager.cc:1397] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 15:57:23.866607 26069 raft_consensus.cc:348] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.866907 26069 raft_consensus.cc:374] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.867026 26069 raft_consensus.cc:729] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4945197a85248daa2e963d97512307c, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.867385 26069 consensus_queue.cc:260] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.868018 26069 ts_tablet_manager.cc:1428] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:57:23.867859 26076 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:23.868264 26076 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:23.868983 26070 tablet_bootstrap.cc:492] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: No bootstrap required, opened a new log
I20260929 15:57:23.869215 26070 ts_tablet_manager.cc:1397] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260929 15:57:23.870095 26070 raft_consensus.cc:348] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.870361 26070 raft_consensus.cc:374] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.870447 26070 raft_consensus.cc:729] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9d95238185425da7120f39dd62a64f, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.870728 26070 consensus_queue.cc:260] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.871390 26070 ts_tablet_manager.cc:1428] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:57:23.871225 26080 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:23.871600 26080 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:23.872627 26071 tablet_bootstrap.cc:492] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: No bootstrap required, opened a new log
I20260929 15:57:23.872795 26071 ts_tablet_manager.cc:1397] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20260929 15:57:23.873490 26071 raft_consensus.cc:348] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.873723 26071 raft_consensus.cc:374] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:23.873790 26071 raft_consensus.cc:729] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30e5427bd0e463fa29d776eb7dc3890, State: Initialized, Role: FOLLOWER
I20260929 15:57:23.874037 26071 consensus_queue.cc:260] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.874591 26075 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:23.874972 26075 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:23.875502 26071 ts_tablet_manager.cc:1428] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:23.920454 26075 raft_consensus.cc:482] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:23.920709 26075 raft_consensus.cc:504] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.922516 26075 leader_election.cc:288] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f9d95238185425da7120f39dd62a64f (127.23.222.65:41679), a4945197a85248daa2e963d97512307c (127.23.222.67:36655)
I20260929 15:57:23.931200 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.923020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:46653 (local address 127.23.222.67:36655)
0929 15:57:23.923202 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.923208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.924230 (+ 1022us) server_negotiation.cc:407] Connection header received
0929 15:57:23.924506 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.924512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.924596 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.924663 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:23.925768 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.926910 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.929382 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.930145 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.930325 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.930402 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.930753 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.930895 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:57:23.931565 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.922896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:41679 (local address 127.23.222.66:56827)
0929 15:57:23.923704 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.923786 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.924036 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.924565 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.924569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.924600 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:23.925073 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.925087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.926637 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.926644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.929007 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.929021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.930723 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.930729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.930878 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.930907 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.930995 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.931123 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":94,"thread_start_us":476,"threads_started":1}
I20260929 15:57:23.931533 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.923302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:56827 (local address 127.23.222.65:41679)
0929 15:57:23.923523 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.923529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.923882 (+ 353us) server_negotiation.cc:407] Connection header received
0929 15:57:23.924215 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.924220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.924297 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.924360 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:23.925666 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.926497 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.929179 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.930478 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.930660 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.930734 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.931040 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.931174 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:23.931972 26112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.922907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:36655 (local address 127.23.222.66:46653)
0929 15:57:23.924122 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.924139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.924340 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.924857 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.924861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.924884 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:23.925444 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.925454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.927082 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.927087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.929240 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.929249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.930352 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.930359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.930535 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.930559 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.931421 (+ 862us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.931587 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"mutex_wait_us":257,"thread_start_us":621,"threads_started":1}
I20260929 15:57:23.932525 25726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f2a931119c41b08a6cd10db41ec922" candidate_uuid: "e30e5427bd0e463fa29d776eb7dc3890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9d95238185425da7120f39dd62a64f" is_pre_election: true
I20260929 15:57:23.932592 26005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f2a931119c41b08a6cd10db41ec922" candidate_uuid: "e30e5427bd0e463fa29d776eb7dc3890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4945197a85248daa2e963d97512307c" is_pre_election: true
I20260929 15:57:23.932802 25726 raft_consensus.cc:2454] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30e5427bd0e463fa29d776eb7dc3890 in term 0.
I20260929 15:57:23.932848 26005 raft_consensus.cc:2454] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e30e5427bd0e463fa29d776eb7dc3890 in term 0.
I20260929 15:57:23.933369 25793 leader_election.cc:302] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9f9d95238185425da7120f39dd62a64f, a4945197a85248daa2e963d97512307c, e30e5427bd0e463fa29d776eb7dc3890; no voters:
I20260929 15:57:23.933822 26075 raft_consensus.cc:2781] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:23.933964 26075 raft_consensus.cc:482] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:23.934064 26075 raft_consensus.cc:3037] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:23.936324 26075 raft_consensus.cc:504] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.936918 26075 leader_election.cc:288] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [CANDIDATE]: Term 1 election: Requested vote from peers 9f9d95238185425da7120f39dd62a64f (127.23.222.65:41679), a4945197a85248daa2e963d97512307c (127.23.222.67:36655)
I20260929 15:57:23.937413 25726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f2a931119c41b08a6cd10db41ec922" candidate_uuid: "e30e5427bd0e463fa29d776eb7dc3890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9d95238185425da7120f39dd62a64f"
I20260929 15:57:23.937453 26005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f2a931119c41b08a6cd10db41ec922" candidate_uuid: "e30e5427bd0e463fa29d776eb7dc3890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4945197a85248daa2e963d97512307c"
I20260929 15:57:23.937646 25726 raft_consensus.cc:3037] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:23.937677 26005 raft_consensus.cc:3037] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:23.940333 25726 raft_consensus.cc:2454] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30e5427bd0e463fa29d776eb7dc3890 in term 1.
I20260929 15:57:23.940742 25793 leader_election.cc:302] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f9d95238185425da7120f39dd62a64f, e30e5427bd0e463fa29d776eb7dc3890; no voters:
I20260929 15:57:23.941098 26075 raft_consensus.cc:2781] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:23.941220 26005 raft_consensus.cc:2454] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e30e5427bd0e463fa29d776eb7dc3890 in term 1.
I20260929 15:57:23.941640 26075 raft_consensus.cc:686] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [term 1 LEADER]: Becoming Leader. State: Replica: e30e5427bd0e463fa29d776eb7dc3890, State: Running, Role: LEADER
I20260929 15:57:23.942092 26075 consensus_queue.cc:237] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } }
I20260929 15:57:23.945197 26113 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: TxnStatusTablet state changed. Reason: New leader e30e5427bd0e463fa29d776eb7dc3890. Latest consensus state: current_term: 1 leader_uuid: "e30e5427bd0e463fa29d776eb7dc3890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:23.945523 26113 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:23.946238 26115 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:57:23.945863 25563 catalog_manager.cc:5495] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 reported cstate change: term changed from 0 to 1, leader changed from <none> to e30e5427bd0e463fa29d776eb7dc3890 (127.23.222.66). New cstate: current_term: 1 leader_uuid: "e30e5427bd0e463fa29d776eb7dc3890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } health_report { overall_health: UNKNOWN } } }
I20260929 15:57:23.946545 26115 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:57:23.981968 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.976754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52248 (local address 127.23.222.66:35113)
0929 15:57:23.976938 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:23.976945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:23.976965 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:57:23.977056 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:23.977059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:23.977158 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:23.977302 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:23.977660 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.978427 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.979756 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.980197 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.980334 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.980773 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:23.981269 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:23.981433 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:23.981551 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 15:57:23.981634 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:57:23.981952 26107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:23.976454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35113 (local address 127.0.0.1:52248)
0929 15:57:23.976659 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:23.976680 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:23.976816 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:23.977325 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:23.977329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:23.977340 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:23.977522 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.977530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.978561 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.978569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.979617 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:23.979629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:23.980349 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:23.980355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:23.980518 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:23.980569 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:23.981387 (+ 818us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:23.981417 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 15:57:23.981506 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 15:57:23.981655 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 15:57:23.986130 26005 raft_consensus.cc:1261] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c [term 1 FOLLOWER]: Refusing update from remote peer e30e5427bd0e463fa29d776eb7dc3890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:23.986528 25726 raft_consensus.cc:1261] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f [term 1 FOLLOWER]: Refusing update from remote peer e30e5427bd0e463fa29d776eb7dc3890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:23.987041 26075 consensus_queue.cc:1035] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:23.987385 26113 consensus_queue.cc:1035] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:23.994091 26076 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: TxnStatusTablet state changed. Reason: New leader e30e5427bd0e463fa29d776eb7dc3890. Latest consensus state: current_term: 1 leader_uuid: "e30e5427bd0e463fa29d776eb7dc3890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:23.994501 26076 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:23.999472 26113 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e30e5427bd0e463fa29d776eb7dc3890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:23.999763 26113 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:24.000099 26085 mvcc.cc:205] Tried to move back new op lower bound from 7334696730559295488 to 7334696730390872064. Current Snapshot: MvccSnapshot[applied={T|T < 7334696730559295488}]
I20260929 15:57:24.000677 26113 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e30e5427bd0e463fa29d776eb7dc3890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:24.000924 26113 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P e30e5427bd0e463fa29d776eb7dc3890: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:24.003033 26084 mvcc.cc:205] Tried to move back new op lower bound from 7334696730559295488 to 7334696730390872064. Current Snapshot: MvccSnapshot[applied={T|T < 7334696730559295488 or (T in {7334696730559295488})}]
I20260929 15:57:24.003430 26076 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e30e5427bd0e463fa29d776eb7dc3890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:24.003719 26076 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P a4945197a85248daa2e963d97512307c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:24.006238 26080 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: TxnStatusTablet state changed. Reason: New leader e30e5427bd0e463fa29d776eb7dc3890. Latest consensus state: current_term: 1 leader_uuid: "e30e5427bd0e463fa29d776eb7dc3890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:24.006580 26080 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:24.009856 26080 tablet_replica.cc:439] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e30e5427bd0e463fa29d776eb7dc3890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 } } peers { permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 } } peers { permanent_uuid: "a4945197a85248daa2e963d97512307c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36655 } } }
I20260929 15:57:24.010150 26080 tablet_replica.cc:442] T b8f2a931119c41b08a6cd10db41ec922 P 9f9d95238185425da7120f39dd62a64f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:24.016543 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.010443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42496 (local address 127.23.222.67:36655)
0929 15:57:24.010624 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.010630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.010648 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:57:24.010759 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.010764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.010886 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.011034 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:24.011797 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.012768 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.014239 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.014645 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.014785 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.015404 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:24.015930 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:24.016100 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.016158 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.016237 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 15:57:24.032913 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.026390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:42945 (local address 127.23.222.66:35113)
0929 15:57:24.026626 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.026636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.027896 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 15:57:24.028111 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.028117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.028195 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.028297 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:24.028781 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.029545 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.030920 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.031467 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.031646 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.032115 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:24.032218 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:24.032318 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.032462 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.032553 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:57:24.033321 26122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.026382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35113 (local address 127.23.222.67:42945)
0929 15:57:24.027785 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:24.027805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:24.027965 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:24.028309 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:24.028313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:24.028324 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:24.028614 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:24.028622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.029677 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.029685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:24.030774 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:24.030786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.031670 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.031676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:24.031841 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.031896 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:24.032320 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:24.032334 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 15:57:24.032513 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 15:57:24.032650 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"thread_start_us":1074,"threads_started":1}
I20260929 15:57:24.045043 25860 raft_consensus.cc:1261] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Refusing update from remote peer a4945197a85248daa2e963d97512307c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:24.045043 25726 raft_consensus.cc:1261] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [term 1 FOLLOWER]: Refusing update from remote peer a4945197a85248daa2e963d97512307c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:24.045746 26100 consensus_queue.cc:1035] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30e5427bd0e463fa29d776eb7dc3890" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:24.046135 26076 consensus_queue.cc:1035] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9d95238185425da7120f39dd62a64f" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:24.093500 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.085682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:58945 (local address 127.23.222.67:36655)
0929 15:57:24.085873 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.085880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.086635 (+ 755us) server_negotiation.cc:407] Connection header received
0929 15:57:24.086842 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.086847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.086962 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.087040 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:24.087718 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.088789 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.090911 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.091699 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.091819 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.092684 (+ 865us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:24.092825 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:24.092887 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.093126 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.093207 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 15:57:24.094188 26126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.085772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:36655 (local address 127.23.222.66:58945)
0929 15:57:24.086530 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:24.086550 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:24.086709 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:24.087067 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:24.087071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:24.087083 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:24.087537 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:24.087550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.088989 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.088993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:24.090779 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:24.090792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.092205 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.092217 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:24.092402 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.092476 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:24.092968 (+ 492us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:24.092978 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:57:24.093757 (+ 779us) client_negotiation.cc:240] Negotiation successful
0929 15:57:24.093882 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":413,"threads_started":1}
I20260929 15:57:24.153532 25565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:24.153798 25565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:24.157691 25565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac8e63f6a15d4096aea8fd6127cd778e: Sending DeleteTablet for 3 replicas of tablet ee9751312ac24d4ba9e51d79e757559f
I20260929 15:57:24.158805 25980 tablet_service.cc:1514] Processing DeleteTablet for tablet ee9751312ac24d4ba9e51d79e757559f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:24 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:42484
I20260929 15:57:24.159071 26093 tablet_replica.cc:330] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: stopping tablet replica
I20260929 15:57:24.159302 26093 raft_consensus.cc:2229] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:24.159847 26093 raft_consensus.cc:2258] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:24.159845 25841 tablet_service.cc:1514] Processing DeleteTablet for tablet ee9751312ac24d4ba9e51d79e757559f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:24 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:52240
I20260929 15:57:24.160296 26095 tablet_replica.cc:330] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: stopping tablet replica
I20260929 15:57:24.160508 26095 raft_consensus.cc:2229] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:24.160578 25702 tablet_service.cc:1514] Processing DeleteTablet for tablet ee9751312ac24d4ba9e51d79e757559f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:24 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:58860
I20260929 15:57:24.160712 26095 raft_consensus.cc:2258] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:24.160993 26094 tablet_replica.cc:330] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: stopping tablet replica
I20260929 15:57:24.161183 26094 raft_consensus.cc:2229] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:24.161403 26094 raft_consensus.cc:2258] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:24.164139 26095 ts_tablet_manager.cc:1905] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:24.172474 26093 ts_tablet_manager.cc:1905] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:24.173593 26094 ts_tablet_manager.cc:1905] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:24.181265 26094 ts_tablet_manager.cc:1918] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:24.181442 26094 log.cc:1196] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/ee9751312ac24d4ba9e51d79e757559f
I20260929 15:57:24.181871 26094 ts_tablet_manager.cc:1939] T ee9751312ac24d4ba9e51d79e757559f P 9f9d95238185425da7120f39dd62a64f: Deleting consensus metadata
I20260929 15:57:24.183068 25550 catalog_manager.cc:4841] TS 9f9d95238185425da7120f39dd62a64f (127.23.222.65:41679): tablet ee9751312ac24d4ba9e51d79e757559f (table test-table [id=72b3a8eb560948569035f9a5344a581d]) successfully deleted
I20260929 15:57:24.193715 26095 ts_tablet_manager.cc:1918] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:24.193929 26095 log.cc:1196] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/ee9751312ac24d4ba9e51d79e757559f
I20260929 15:57:24.194406 26095 ts_tablet_manager.cc:1939] T ee9751312ac24d4ba9e51d79e757559f P e30e5427bd0e463fa29d776eb7dc3890: Deleting consensus metadata
I20260929 15:57:24.195124 26093 ts_tablet_manager.cc:1918] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:24.195348 26093 log.cc:1196] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/ee9751312ac24d4ba9e51d79e757559f
I20260929 15:57:24.195684 25552 catalog_manager.cc:4841] TS e30e5427bd0e463fa29d776eb7dc3890 (127.23.222.66:35113): tablet ee9751312ac24d4ba9e51d79e757559f (table test-table [id=72b3a8eb560948569035f9a5344a581d]) successfully deleted
I20260929 15:57:24.195818 26093 ts_tablet_manager.cc:1939] T ee9751312ac24d4ba9e51d79e757559f P a4945197a85248daa2e963d97512307c: Deleting consensus metadata
I20260929 15:57:24.207173 25551 catalog_manager.cc:4841] TS a4945197a85248daa2e963d97512307c (127.23.222.67:36655): tablet ee9751312ac24d4ba9e51d79e757559f (table test-table [id=72b3a8eb560948569035f9a5344a581d]) successfully deleted
Sep 29 15:57:24 dist-test-slave-cq88 krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:24.231787 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.172241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.23.222.65:41679)
0929 15:57:24.172445 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.172451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.211362 (+ 38911us) server_negotiation.cc:407] Connection header received
0929 15:57:24.211753 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.211760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.211851 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.211932 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:24.214140 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.215273 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.217236 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.217995 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.218098 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.223715 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:24.223738 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:24.223763 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:24.223814 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:24.228072 (+ 4258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.229004 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.229014 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.229018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.229196 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.229687 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.229693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.229696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.230161 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:24.230482 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.231092 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.231370 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 15:57:24.233371 25767 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58880
I20260929 15:57:24.264572 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.244478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.23.222.126:34615)
0929 15:57:24.244679 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.244685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.247838 (+ 3153us) server_negotiation.cc:407] Connection header received
0929 15:57:24.248177 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.248183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.248280 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.248359 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:24.250359 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.251550 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.253509 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.254191 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.254324 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.257574 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:24.257581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:24.257587 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:24.257630 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:24.261321 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.262207 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.262213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.262217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.262366 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.262826 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.262831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.262835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.263223 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:24.263584 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.263909 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.264145 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 15:57:24.265144 25565 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53936
I20260929 15:57:24.267925 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25636
I20260929 15:57:24.285275 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.269857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.23.222.65:41679)
0929 15:57:24.270042 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.270048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.270068 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:57:24.270277 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.270283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.270368 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.270455 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:24.272176 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.273266 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.275090 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.275786 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.275908 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.278747 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:24.278754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:24.278759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:24.278801 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:24.282189 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.282972 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.282978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.282982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.283136 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.283614 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.283620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.283623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.283948 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:24.284207 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.284508 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.284738 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 15:57:24.285825 25767 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58886
W20260929 15:57:24.286459 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25636: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:24.286661 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 25636
I20260929 15:57:24.321257 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25776
Sep 29 15:57:24 dist-test-slave-cq88 krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:57:24.343514 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.323023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52258 (local address 127.23.222.66:35113)
0929 15:57:24.323260 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.323268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.323289 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:57:24.323357 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.323363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.323466 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.323589 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:24.325237 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.326648 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.328552 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.329401 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.329563 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.335225 (+ 5662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:24.335263 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:24.335287 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:24.335357 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:24.339718 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.340780 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.340790 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.340794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.341025 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.341514 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.341519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.341523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.341974 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:24.342309 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.342642 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.342879 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 15:57:24.344331 25906 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52258
W20260929 15:57:24.345134 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25776: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:24.345321 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 25776
I20260929 15:57:24.371856 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25915
Sep 29 15:57:24 dist-test-slave-cq88 krb5kdc[25520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697443, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:57:24.391628 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:24.372649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.23.222.67:36655)
0929 15:57:24.372832 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:24.372838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:24.372955 (+ 117us) server_negotiation.cc:407] Connection header received
0929 15:57:24.373146 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:24.373152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:24.373226 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:24.373335 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:24.374937 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.376112 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.377952 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:24.378646 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:24.378764 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:24.384119 (+ 5355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:24.384134 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:24.384158 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:24.384206 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:24.388449 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.389223 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.389233 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.389237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.389409 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:24.389889 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:24.389894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:24.389898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:24.390374 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:24.390741 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:24.390964 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 15:57:24.391187 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 15:57:24.392303 26045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42508
W20260929 15:57:24.392990 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25915: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:24.393141 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 25915
I20260929 15:57:24.412518 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 25533
I20260929 15:57:24.413740 25605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:24.530797 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 25533
2026-09-29T15:57:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4576 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:24 dist-test-slave-cq88 krb5kdc[26137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:24 dist-test-slave-cq88 krb5kdc[26137](info): set up 2 sockets
Sep 29 15:57:24 dist-test-slave-cq88 krb5kdc[26137](info): commencing operation
krb5kdc: starting...
W20260929 15:57:25.622365 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:57:25 dist-test-slave-cq88 krb5kdc[26137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:57:25.811206 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34507
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:34507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:26.129520 26150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:26.129884 26150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:26.130059 26150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:26.137507 26150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:26.137667 26150 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:57:26.137763 26150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:26.137881 26150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:26.137967 26150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:26.138056 26150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:26.148226 26150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:34507
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:26.149611 26150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:26.150996 26150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:26.163636 26158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:26.163654 26156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:26.164004 26155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:26.164848 26150 server_base.cc:1034] running on GCE node
I20260929 15:57:26.165699 26150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:26.167141 26150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:26.168395 26150 hybrid_clock.cc:648] HybridClock initialized: now 1790697446168327 us; error 27 us; skew 500 ppm
I20260929 15:57:26.168716 26150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:26 dist-test-slave-cq88 krb5kdc[26137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:26.171875 26150 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:57:26.173348 26150 webserver.cc:460] Webserver started at http://127.23.222.126:35303/ using document root <none> and password file <none>
I20260929 15:57:26.174037 26150 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:26.174124 26150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:26.174419 26150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:26.177474 26150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "519f3739025a4ba9aafac43aa730d0f9"
format_stamp: "Formatted at 2026-09-29 15:57:26 on dist-test-slave-cq88"
server_key: "9f908460b736966f302a7971c3d1ca39"
server_key_iv: "44c611381941aded7cfca42d409f6671"
server_key_version: "encryptionkey@0"
I20260929 15:57:26.178494 26150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "519f3739025a4ba9aafac43aa730d0f9"
format_stamp: "Formatted at 2026-09-29 15:57:26 on dist-test-slave-cq88"
server_key: "9f908460b736966f302a7971c3d1ca39"
server_key_iv: "44c611381941aded7cfca42d409f6671"
server_key_version: "encryptionkey@0"
I20260929 15:57:26.186025 26150 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.001s
I20260929 15:57:26.189644 26165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:26.191030 26150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:26.191174 26150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "519f3739025a4ba9aafac43aa730d0f9"
format_stamp: "Formatted at 2026-09-29 15:57:26 on dist-test-slave-cq88"
server_key: "9f908460b736966f302a7971c3d1ca39"
server_key_iv: "44c611381941aded7cfca42d409f6671"
server_key_version: "encryptionkey@0"
I20260929 15:57:26.191459 26150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:26.210969 26150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:26.215649 26150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:26.215884 26150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:26.238205 26150 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:34507
I20260929 15:57:26.238292 26226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:34507 every 8 connection(s)
I20260929 15:57:26.239725 26150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:26.243626 26227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:26.248656 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 26150
I20260929 15:57:26.248869 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:26.249475 24441 external_mini_cluster.cc:1428] Setting key b5baae4a9d1cbc451a00535be9fbe013
I20260929 15:57:26.254252 26227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9: Bootstrap starting.
I20260929 15:57:26.257949 26227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:26.259227 26227 log.cc:824] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:26.262691 26227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9: No bootstrap required, opened a new log
Sep 29 15:57:26 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:26.270777 26227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519f3739025a4ba9aafac43aa730d0f9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34507 } }
I20260929 15:57:26.271193 26227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:26.271368 26227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519f3739025a4ba9aafac43aa730d0f9, State: Initialized, Role: FOLLOWER
I20260929 15:57:26.272130 26227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519f3739025a4ba9aafac43aa730d0f9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34507 } }
I20260929 15:57:26.272392 26227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:26.272557 26227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:26.272768 26227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:26.273343 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:26.251472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.23.222.126:34507)
0929 15:57:26.252299 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:26.252314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:26.252358 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:57:26.253967 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:26.253989 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:26.254429 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:26.254766 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:26.256659 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.257912 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.259687 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.260793 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.260931 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:26.265689 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:26.265702 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:26.265721 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:26.265768 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:26.268999 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:26.269726 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:26.269733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:26.269743 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:26.269913 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:26.270331 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:26.270339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:26.270347 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:26.270882 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:26.271199 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:26.271658 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 15:57:26.272092 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":396,"threads_started":1}
I20260929 15:57:26.275815 26227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519f3739025a4ba9aafac43aa730d0f9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34507 } }
I20260929 15:57:26.276398 26227 leader_election.cc:302] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519f3739025a4ba9aafac43aa730d0f9; no voters:
I20260929 15:57:26.276974 26227 leader_election.cc:288] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:26.277132 26232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:26.277834 26232 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [term 1 LEADER]: Becoming Leader. State: Replica: 519f3739025a4ba9aafac43aa730d0f9, State: Running, Role: LEADER
I20260929 15:57:26.278333 26232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519f3739025a4ba9aafac43aa730d0f9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34507 } }
I20260929 15:57:26.279028 26227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:26.281348 26233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "519f3739025a4ba9aafac43aa730d0f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519f3739025a4ba9aafac43aa730d0f9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34507 } } }
I20260929 15:57:26.281703 26233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:26.282548 26238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:26.283483 26234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519f3739025a4ba9aafac43aa730d0f9. Latest consensus state: current_term: 1 leader_uuid: "519f3739025a4ba9aafac43aa730d0f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519f3739025a4ba9aafac43aa730d0f9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34507 } } }
I20260929 15:57:26.283852 26234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:26.287930 26238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:26.297894 26238 catalog_manager.cc:1347] Generated new cluster ID: f3fb524585ef4224b492bf26d9e65876
I20260929 15:57:26.298071 26238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:26.319079 26238 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:26.321110 26238 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:26.333285 26238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9: Generated new TSK 0
I20260929 15:57:26.334200 26238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:57:26.388029 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34507
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:26.715973 26253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:26.716266 26253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:26.716370 26253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:26.724068 26253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:26.724193 26253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:26.724279 26253 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:57:26.724334 26253 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:26.724396 26253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:57:26.734328 26253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.222.126:34507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:26.735730 26253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:26.737003 26253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:26.750347 26258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:26.750300 26261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:26.750319 26259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:26.751319 26253 server_base.cc:1034] running on GCE node
I20260929 15:57:26.752031 26253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:26.752980 26253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:26.754215 26253 hybrid_clock.cc:648] HybridClock initialized: now 1790697446754179 us; error 56 us; skew 500 ppm
I20260929 15:57:26.754555 26253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:26 dist-test-slave-cq88 krb5kdc[26137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:26.758189 26253 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:57:26.759835 26253 webserver.cc:460] Webserver started at http://127.23.222.65:38119/ using document root <none> and password file <none>
I20260929 15:57:26.760505 26253 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:26.760667 26253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:26.761036 26253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:26.764099 26253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "7ff703e4a46b4360af610690bf372514"
format_stamp: "Formatted at 2026-09-29 15:57:26 on dist-test-slave-cq88"
server_key: "5e8f41b6c3cef1f8912ce45d8664f62f"
server_key_iv: "8c2a82347c1a80c24d1798f8ca817862"
server_key_version: "encryptionkey@0"
I20260929 15:57:26.764873 26253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "7ff703e4a46b4360af610690bf372514"
format_stamp: "Formatted at 2026-09-29 15:57:26 on dist-test-slave-cq88"
server_key: "5e8f41b6c3cef1f8912ce45d8664f62f"
server_key_iv: "8c2a82347c1a80c24d1798f8ca817862"
server_key_version: "encryptionkey@0"
I20260929 15:57:26.770370 26253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:57:26.774546 26268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:26.776141 26253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:26.776288 26253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "7ff703e4a46b4360af610690bf372514"
format_stamp: "Formatted at 2026-09-29 15:57:26 on dist-test-slave-cq88"
server_key: "5e8f41b6c3cef1f8912ce45d8664f62f"
server_key_iv: "8c2a82347c1a80c24d1798f8ca817862"
server_key_version: "encryptionkey@0"
I20260929 15:57:26.776464 26253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:26.796867 26253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:26.801683 26253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:26.801946 26253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:26.804925 26253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:26.805068 26253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:26.805198 26253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:26.805300 26253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:26 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:26.831794 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:26.805474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:37215 (local address 127.23.222.126:34507)
0929 15:57:26.806317 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:26.806322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:26.807335 (+ 1013us) server_negotiation.cc:407] Connection header received
0929 15:57:26.809457 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:26.809463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:26.809567 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:26.809692 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:26.812219 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.813247 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.816866 (+ 3619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.817429 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.817611 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:26.823982 (+ 6371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:26.823988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:26.823992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:26.824031 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:26.827562 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:26.828769 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:26.828778 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:26.828785 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:26.828992 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:26.829648 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:26.829653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:26.829657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:26.830111 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:26.830272 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:26.831134 (+ 862us) server_negotiation.cc:299] Negotiation successful
0929 15:57:26.831419 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":627,"threads_started":1}
I20260929 15:57:26.833182 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:26.806258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34507 (local address 127.23.222.65:37215)
0929 15:57:26.807128 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:26.807169 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:26.809118 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:26.809970 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:26.809979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:26.810944 (+ 965us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:26.811994 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:26.812019 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.813506 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.813515 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:26.816557 (+ 3042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:26.816581 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.817729 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.817740 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:26.818198 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:26.819404 (+ 1206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:26.819486 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:26.823560 (+ 4074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:26.827805 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:26.827815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:26.827850 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:26.828581 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:26.829135 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:26.829143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:26.829148 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:26.829511 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:26.830252 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:26.830261 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:26.830702 (+ 441us) client_negotiation.cc:769] Sending connection context
0929 15:57:26.831040 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 15:57:26.831729 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":360,"threads_started":1}
I20260929 15:57:26.858414 26253 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:36807
I20260929 15:57:26.858459 26388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:36807 every 8 connection(s)
I20260929 15:57:26.860164 26253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:26.861183 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:26.846912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34507 (local address 127.23.222.65:50061)
0929 15:57:26.847765 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:26.847792 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:26.848000 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:26.848388 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:26.848393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:26.848906 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:26.849196 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:26.849206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.850203 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.850210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:26.851333 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:26.851345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.852295 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.852307 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:26.852491 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:26.853339 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:26.853356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:26.854430 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:26.858228 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:26.858233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:26.858237 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:26.858828 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:26.859266 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:26.859273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:26.859276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:26.859424 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:26.859959 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:26.859963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:26.860167 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 15:57:26.860328 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 15:57:26.860597 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":430,"threads_started":1}
I20260929 15:57:26.861259 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:26.846928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:50061 (local address 127.23.222.126:34507)
0929 15:57:26.847187 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:26.847197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:26.847922 (+ 725us) server_negotiation.cc:407] Connection header received
0929 15:57:26.848138 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:26.848143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:26.848211 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:26.848350 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:26.849353 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.850060 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.851471 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:26.852079 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:26.852216 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:26.854637 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:26.854642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:26.854645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:26.854679 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:26.858066 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:26.858967 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:26.858973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:26.858976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:26.859116 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:26.859547 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:26.859554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:26.859561 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:26.859836 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:26.859995 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:26.860503 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 15:57:26.860804 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:57:26.863345 26389 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34507
I20260929 15:57:26.863598 26389 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:26.864476 26389 heartbeater.cc:511] Master 127.23.222.126:34507 requested a full tablet report, sending...
I20260929 15:57:26.866640 26182 ts_manager.cc:194] Registered new tserver with Master: 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807)
I20260929 15:57:26.867040 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 26253
I20260929 15:57:26.867280 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:26.867903 24441 external_mini_cluster.cc:1428] Setting key 74a56b9ce9e4dbd2bb06ce77ac4edc05
I20260929 15:57:26.868328 26182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:37215
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:57:26.911767 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34507
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:27.229972 26393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:27.230312 26393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:27.230504 26393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:27.238113 26393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:27.238261 26393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:27.238432 26393 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:57:27.238523 26393 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:27.238662 26393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:57:27.248252 26393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.222.126:34507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:27.249549 26393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:27.250850 26393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:27.263680 26401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:27.263618 26399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:27.263759 26398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:27.264822 26393 server_base.cc:1034] running on GCE node
I20260929 15:57:27.265405 26393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:27.266346 26393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:27.267550 26393 hybrid_clock.cc:648] HybridClock initialized: now 1790697447267508 us; error 26 us; skew 500 ppm
I20260929 15:57:27.267868 26393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:27 dist-test-slave-cq88 krb5kdc[26137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:27.270946 26393 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:57:27.272394 26393 webserver.cc:460] Webserver started at http://127.23.222.66:35837/ using document root <none> and password file <none>
I20260929 15:57:27.273080 26393 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:27.273224 26393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:27.273540 26393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:27.276597 26393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "dbc0949aa70b485096df5ce8f17dbdb2"
format_stamp: "Formatted at 2026-09-29 15:57:27 on dist-test-slave-cq88"
server_key: "5e5962dba7fad184986778a8445dd04f"
server_key_iv: "23e1816fba90a6d984393d80f66b4984"
server_key_version: "encryptionkey@0"
I20260929 15:57:27.277376 26393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "dbc0949aa70b485096df5ce8f17dbdb2"
format_stamp: "Formatted at 2026-09-29 15:57:27 on dist-test-slave-cq88"
server_key: "5e5962dba7fad184986778a8445dd04f"
server_key_iv: "23e1816fba90a6d984393d80f66b4984"
server_key_version: "encryptionkey@0"
I20260929 15:57:27.282786 26393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 15:57:27.286114 26408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:27.287478 26393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:57:27.287647 26393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "dbc0949aa70b485096df5ce8f17dbdb2"
format_stamp: "Formatted at 2026-09-29 15:57:27 on dist-test-slave-cq88"
server_key: "5e5962dba7fad184986778a8445dd04f"
server_key_iv: "23e1816fba90a6d984393d80f66b4984"
server_key_version: "encryptionkey@0"
I20260929 15:57:27.287891 26393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:27.310381 26393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:27.314237 26393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:27.314532 26393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:27.317119 26393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:27.317261 26393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:27.317389 26393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:27.317482 26393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:57:27 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:27.342834 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.317846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:52809 (local address 127.23.222.126:34507)
0929 15:57:27.318057 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:27.318063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:27.319538 (+ 1475us) server_negotiation.cc:407] Connection header received
0929 15:57:27.321630 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:27.321637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:27.321738 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:27.321859 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:27.324710 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.325661 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.328998 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.329603 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.329770 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.335056 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:27.335065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:27.335072 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:27.335122 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:27.339283 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.340330 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.340337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.340343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.340533 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.341039 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.341044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.341048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.341348 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:27.341546 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:27.342261 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 15:57:27.342496 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:57:27.344280 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.318427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34507 (local address 127.23.222.66:52809)
0929 15:57:27.319338 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:27.319387 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:27.321277 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:27.322292 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:27.322307 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:27.323331 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:27.324461 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.324499 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.325849 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.325860 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.328756 (+ 2896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.328776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.329893 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.329903 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.330283 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.331223 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:27.331287 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:27.334762 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:27.339456 (+ 4694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:27.339465 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:27.339491 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:27.340177 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:27.340657 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:27.340660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:27.340663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:27.340886 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:27.341482 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:27.341492 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:27.341880 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 15:57:27.342157 (+ 277us) client_negotiation.cc:240] Negotiation successful
0929 15:57:27.342689 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":384,"threads_started":1}
I20260929 15:57:27.367202 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.351310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:34279 (local address 127.23.222.126:34507)
0929 15:57:27.351551 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:27.351560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:27.352247 (+ 687us) server_negotiation.cc:407] Connection header received
0929 15:57:27.352529 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:27.352537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:27.352626 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:27.352760 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:27.354206 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.355450 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.357551 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.358154 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.358287 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.360618 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:27.360623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:27.360627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:27.360658 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:27.363762 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.364803 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.364812 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.364816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.365030 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.365558 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.365567 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.365571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.365953 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:27.366077 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:27.366626 (+ 549us) server_negotiation.cc:299] Negotiation successful
0929 15:57:27.366896 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 15:57:27.367648 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.351368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34507 (local address 127.23.222.66:34279)
0929 15:57:27.352109 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:27.352137 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:27.352369 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:27.352798 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:27.352807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:27.353566 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:27.354022 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.354044 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.355618 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.355622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.357310 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.357332 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.358366 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.358373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.358542 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.359378 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:27.359396 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:27.360404 (+ 1008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:27.363972 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:27.363979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:27.363985 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:27.364652 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:27.365179 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:27.365185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:27.365190 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:27.365395 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:27.366072 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:27.366093 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:27.366286 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 15:57:27.366463 (+ 177us) client_negotiation.cc:240] Negotiation successful
0929 15:57:27.366956 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":420,"threads_started":1}
I20260929 15:57:27.372423 26393 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:35849
I20260929 15:57:27.372671 26527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:35849 every 8 connection(s)
I20260929 15:57:27.373805 26393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:27.375885 26528 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34507
I20260929 15:57:27.376104 26528 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:27.376706 26528 heartbeater.cc:511] Master 127.23.222.126:34507 requested a full tablet report, sending...
I20260929 15:57:27.377996 26181 ts_manager.cc:194] Registered new tserver with Master: dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849)
I20260929 15:57:27.379091 26181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:52809
I20260929 15:57:27.379391 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 26393
I20260929 15:57:27.379575 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:27.380206 24441 external_mini_cluster.cc:1428] Setting key 747348f18dd0fbaeb24d52826e77fa65
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:57:27.422798 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34507
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:27.751153 26532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:27.751428 26532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:27.751529 26532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:27.758586 26532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:27.758689 26532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:27.758773 26532 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 15:57:27.758824 26532 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:57:27.758898 26532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:57:27.768671 26532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.222.126:34507
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:27.770036 26532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:27.771378 26532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:27.784668 26540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:27.784718 26538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:27.784919 26537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:27.785735 26532 server_base.cc:1034] running on GCE node
I20260929 15:57:27.786489 26532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:27.787657 26532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:27.788915 26532 hybrid_clock.cc:648] HybridClock initialized: now 1790697447788862 us; error 36 us; skew 500 ppm
I20260929 15:57:27.789263 26532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:27 dist-test-slave-cq88 krb5kdc[26137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:27.792886 26532 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:57:27.794334 26532 webserver.cc:460] Webserver started at http://127.23.222.67:42669/ using document root <none> and password file <none>
I20260929 15:57:27.795091 26532 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:27.795228 26532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:27.795580 26532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:27.798470 26532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "6ce2ba1c40a248919ee589726ec8e561"
format_stamp: "Formatted at 2026-09-29 15:57:27 on dist-test-slave-cq88"
server_key: "ffa1091b3be059b41d10165e2ca7599c"
server_key_iv: "1383299ba5261fa08b36b2194ca9d813"
server_key_version: "encryptionkey@0"
I20260929 15:57:27.799217 26532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "6ce2ba1c40a248919ee589726ec8e561"
format_stamp: "Formatted at 2026-09-29 15:57:27 on dist-test-slave-cq88"
server_key: "ffa1091b3be059b41d10165e2ca7599c"
server_key_iv: "1383299ba5261fa08b36b2194ca9d813"
server_key_version: "encryptionkey@0"
I20260929 15:57:27.804512 26532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:57:27.807931 26547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:27.809288 26532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:27.809459 26532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "6ce2ba1c40a248919ee589726ec8e561"
format_stamp: "Formatted at 2026-09-29 15:57:27 on dist-test-slave-cq88"
server_key: "ffa1091b3be059b41d10165e2ca7599c"
server_key_iv: "1383299ba5261fa08b36b2194ca9d813"
server_key_version: "encryptionkey@0"
I20260929 15:57:27.809662 26532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:27.825251 26532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:27.829329 26532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:27.829619 26532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:27.832305 26532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:27.832438 26532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:27.832577 26532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:27.832713 26532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 15:57:27 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:27.856140 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.832762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:47497 (local address 127.23.222.126:34507)
0929 15:57:27.832979 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:27.832986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:27.834403 (+ 1417us) server_negotiation.cc:407] Connection header received
0929 15:57:27.836782 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:27.836788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:27.836873 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:27.836992 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:27.839163 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.840317 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.842979 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.843614 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.843749 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.848423 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:27.848434 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:27.848442 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:27.848510 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:27.852297 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.853240 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.853247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.853251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.853429 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.854207 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.854215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.854220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.854562 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:27.854713 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:27.855556 (+ 843us) server_negotiation.cc:299] Negotiation successful
0929 15:57:27.855783 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:57:27.857622 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.833328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34507 (local address 127.23.222.67:47497)
0929 15:57:27.834216 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:27.834271 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:27.836470 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:27.837226 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:27.837248 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:27.838004 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:27.838966 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.838996 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.840477 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.840483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.842798 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.842812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.843837 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.843845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.844196 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.844984 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:27.845004 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:27.848162 (+ 3158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:27.852434 (+ 4272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:27.852443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:27.852459 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:27.853090 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:27.853601 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:27.853612 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:27.853619 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:27.853986 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:27.854709 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:27.854717 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:27.855084 (+ 367us) client_negotiation.cc:769] Sending connection context
0929 15:57:27.855395 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 15:57:27.856004 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":393,"threads_started":1}
I20260929 15:57:27.871567 26389 heartbeater.cc:503] Master 127.23.222.126:34507 was elected leader, sending a full tablet report...
I20260929 15:57:27.881124 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.867350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:53885 (local address 127.23.222.126:34507)
0929 15:57:27.867563 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:27.867569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:27.868169 (+ 600us) server_negotiation.cc:407] Connection header received
0929 15:57:27.868372 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:27.868379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:27.868457 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:27.868560 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:27.869476 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.870179 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.871608 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.872208 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.872363 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.874771 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:27.874777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:27.874782 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:27.874825 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:27.877932 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.878839 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.878842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.878845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.878961 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.879516 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.879524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.879528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.879872 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:27.880076 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:27.880543 (+ 467us) server_negotiation.cc:299] Negotiation successful
0929 15:57:27.880757 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:27.881212 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.867478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34507 (local address 127.23.222.67:53885)
0929 15:57:27.868063 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:27.868081 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:27.868241 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:27.868583 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:27.868588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:27.869055 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:27.869331 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.869340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.870323 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.870329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.871470 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.871482 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.872404 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.872410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.872582 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.873535 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:27.873552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:27.874577 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:27.878074 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:27.878079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:27.878083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:27.878671 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:27.879147 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:27.879154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:27.879160 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:27.879379 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:27.880032 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:27.880037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:27.880272 (+ 235us) client_negotiation.cc:769] Sending connection context
0929 15:57:27.880414 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 15:57:27.880753 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":336,"threads_started":1}
I20260929 15:57:27.891304 26532 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:38569
I20260929 15:57:27.891454 26666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:38569 every 8 connection(s)
I20260929 15:57:27.892678 26532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:27.894691 26667 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34507
I20260929 15:57:27.894919 26667 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:27.895665 26667 heartbeater.cc:511] Master 127.23.222.126:34507 requested a full tablet report, sending...
I20260929 15:57:27.896974 26181 ts_manager.cc:194] Registered new tserver with Master: 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569)
I20260929 15:57:27.897799 26181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:47497
I20260929 15:57:27.902012 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 26532
I20260929 15:57:27.902199 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:27.902688 24441 external_mini_cluster.cc:1428] Setting key d58b233111ca739e373a3c74068d73b6
I20260929 15:57:27.905725 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:27 dist-test-slave-cq88 krb5kdc[26137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:57:27 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:27.951409 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.937873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.23.222.126:34507)
0929 15:57:27.938070 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:27.938074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:27.938411 (+ 337us) server_negotiation.cc:407] Connection header received
0929 15:57:27.938560 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:27.938564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:27.938632 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:27.938730 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:27.940230 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.940984 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.942199 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.942635 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.942727 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.946330 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:27.946335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:27.946338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:27.946370 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:27.948846 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.949456 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.949469 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.949471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.949583 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:27.949947 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:27.949954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:27.949957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:27.950211 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:27.950463 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:27.950755 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 15:57:27.950985 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:57:27.955907 26181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:27.963116 26181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:27.995903 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.975093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.23.222.66:35849)
0929 15:57:27.987648 (+ 12555us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:27.987656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:27.987685 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:57:27.987805 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:27.987812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:27.988083 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:27.989114 (+ 1031us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:27.989335 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.990841 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.993391 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.994757 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.994886 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.995093 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:27.995398 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 15:57:27.995580 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12378,"thread_start_us":12778,"threads_started":1}
I20260929 15:57:27.996054 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.974961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35849 (local address 127.0.0.1:58170)
0929 15:57:27.986341 (+ 11380us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:27.986398 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:27.986604 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:27.988361 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:27.988369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:27.988399 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:27.988900 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.988932 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.991034 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.991038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.993244 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.993256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.994932 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.994940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.995126 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.995177 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 15:57:27.995332 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 15:57:27.995531 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11049,"mutex_wait_us":99,"spinlock_wait_cycles":4480,"thread_start_us":11834,"threads_started":1}
I20260929 15:57:27.996918 26681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.975437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:38569 (local address 127.0.0.1:40206)
0929 15:57:27.988480 (+ 13043us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:27.988506 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:27.988655 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:27.989242 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:27.989246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:27.989267 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:27.989619 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.989630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.991784 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.991791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.994050 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.994062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.996139 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.996146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.996304 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.996326 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:57:27.996437 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 15:57:27.996569 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12916,"mutex_wait_us":21,"spinlock_wait_cycles":7296,"thread_start_us":10408,"threads_started":1}
I20260929 15:57:27.997013 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.976403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.23.222.67:38569)
0929 15:57:27.983862 (+ 7459us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:27.983875 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:27.988642 (+ 4767us) server_negotiation.cc:407] Connection header received
0929 15:57:27.988824 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:27.988834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:27.989062 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:27.989238 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:27.989845 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.991600 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.994229 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.995634 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.995751 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.995985 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:27.996503 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 15:57:27.996669 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7210,"thread_start_us":7004,"threads_started":1}
I20260929 15:57:27.998149 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.975086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36807 (local address 127.0.0.1:47708)
0929 15:57:27.987125 (+ 12039us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:27.987155 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:27.987344 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:27.988446 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:27.988453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:27.988490 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:27.989075 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.989092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.991029 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.991033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.994317 (+ 3284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:27.994329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.997370 (+ 3041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.997376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:27.997556 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.997587 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:57:27.997669 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 15:57:27.997786 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11877,"mutex_wait_us":69,"spinlock_wait_cycles":22144,"thread_start_us":13095,"threads_started":1}
I20260929 15:57:27.999063 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:27.975886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.23.222.65:36807)
0929 15:57:27.984358 (+ 8472us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:27.984368 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:27.987502 (+ 3134us) server_negotiation.cc:407] Connection header received
0929 15:57:27.987653 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:27.987665 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:27.988013 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:27.988234 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:27.989287 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.990869 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.994492 (+ 3623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:27.996722 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:27.996856 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:27.998508 (+ 1652us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:27.998574 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 15:57:27.998750 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8289,"thread_start_us":7280,"threads_started":1}
I20260929 15:57:28.000216 26458 tablet_service.cc:1467] Processing CreateTablet for tablet 8ababb0962674d36a006719dd711fc67 (DEFAULT_TABLE table=test-table [id=abc96e9eb403446d9f81fbf42aac4bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:28.000531 26597 tablet_service.cc:1467] Processing CreateTablet for tablet 8ababb0962674d36a006719dd711fc67 (DEFAULT_TABLE table=test-table [id=abc96e9eb403446d9f81fbf42aac4bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:28.001621 26597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ababb0962674d36a006719dd711fc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:28.001677 26458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ababb0962674d36a006719dd711fc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:28.002630 26319 tablet_service.cc:1467] Processing CreateTablet for tablet 8ababb0962674d36a006719dd711fc67 (DEFAULT_TABLE table=test-table [id=abc96e9eb403446d9f81fbf42aac4bb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:28.004742 26319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ababb0962674d36a006719dd711fc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:28.012892 26686 tablet_bootstrap.cc:492] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: Bootstrap starting.
I20260929 15:57:28.016027 26687 tablet_bootstrap.cc:492] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: Bootstrap starting.
I20260929 15:57:28.017005 26686 tablet_bootstrap.cc:654] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:28.018393 26686 log.cc:824] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:28.018942 26688 tablet_bootstrap.cc:492] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: Bootstrap starting.
I20260929 15:57:28.020035 26687 tablet_bootstrap.cc:654] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:28.021663 26687 log.cc:824] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:28.021831 26688 tablet_bootstrap.cc:654] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:28.023039 26688 log.cc:824] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:28.024631 26686 tablet_bootstrap.cc:492] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: No bootstrap required, opened a new log
I20260929 15:57:28.024768 26687 tablet_bootstrap.cc:492] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: No bootstrap required, opened a new log
I20260929 15:57:28.024956 26686 ts_tablet_manager.cc:1397] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 15:57:28.025046 26687 ts_tablet_manager.cc:1397] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 15:57:28.026347 26688 tablet_bootstrap.cc:492] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: No bootstrap required, opened a new log
I20260929 15:57:28.026615 26688 ts_tablet_manager.cc:1397] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 15:57:28.032583 26686 raft_consensus.cc:348] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.032997 26686 raft_consensus.cc:374] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:28.033097 26686 raft_consensus.cc:729] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2ba1c40a248919ee589726ec8e561, State: Initialized, Role: FOLLOWER
I20260929 15:57:28.033905 26686 consensus_queue.cc:260] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.035347 26667 heartbeater.cc:503] Master 127.23.222.126:34507 was elected leader, sending a full tablet report...
I20260929 15:57:28.036075 26686 ts_tablet_manager.cc:1428] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260929 15:57:28.035930 26687 raft_consensus.cc:348] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.036369 26687 raft_consensus.cc:374] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:28.036474 26687 raft_consensus.cc:729] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc0949aa70b485096df5ce8f17dbdb2, State: Initialized, Role: FOLLOWER
I20260929 15:57:28.037088 26687 consensus_queue.cc:260] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.038178 26528 heartbeater.cc:503] Master 127.23.222.126:34507 was elected leader, sending a full tablet report...
I20260929 15:57:28.040134 26687 ts_tablet_manager.cc:1428] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260929 15:57:28.039443 26688 raft_consensus.cc:348] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.041036 26688 raft_consensus.cc:374] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:28.041152 26688 raft_consensus.cc:729] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff703e4a46b4360af610690bf372514, State: Initialized, Role: FOLLOWER
I20260929 15:57:28.041783 26688 consensus_queue.cc:260] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.043635 26688 ts_tablet_manager.cc:1428] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
I20260929 15:57:28.063997 26692 raft_consensus.cc:482] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:28.064260 26692 raft_consensus.cc:504] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.065768 26692 leader_election.cc:288] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849), 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807)
I20260929 15:57:28.071905 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.066281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:58095 (local address 127.23.222.66:35849)
0929 15:57:28.066461 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.066470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.066507 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:57:28.066690 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.066698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.066775 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.066879 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:28.067550 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.068643 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.070254 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.071009 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.071178 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.071272 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.071534 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.071655 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:57:28.071985 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.066028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35849 (local address 127.23.222.67:58095)
0929 15:57:28.066322 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.066344 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.066519 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.066924 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.066928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.066954 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:28.067333 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.067345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.068777 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.068784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.070091 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.070102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.071158 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.071164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.071366 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.071390 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.071513 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.071645 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":1,"spinlock_wait_cycles":78720}
I20260929 15:57:28.072196 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.066187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:52015 (local address 127.23.222.65:36807)
0929 15:57:28.066385 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.066391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.067077 (+ 686us) server_negotiation.cc:407] Connection header received
0929 15:57:28.067298 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.067302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.067369 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.067479 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:28.068108 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.068874 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.070515 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.071309 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.071472 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.071547 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.071817 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.071938 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:57:28.072695 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.066134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36807 (local address 127.23.222.67:52015)
0929 15:57:28.066931 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.066955 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.067126 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.067517 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.067521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.067546 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:28.067901 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.067910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.069017 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.069022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.070357 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.070369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.071528 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.071534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.071666 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.071690 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.071792 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.071915 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":439,"threads_started":1}
I20260929 15:57:28.073043 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ababb0962674d36a006719dd711fc67" candidate_uuid: "6ce2ba1c40a248919ee589726ec8e561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" is_pre_election: true
I20260929 15:57:28.073629 26478 raft_consensus.cc:2454] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce2ba1c40a248919ee589726ec8e561 in term 0.
I20260929 15:57:28.073644 26339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ababb0962674d36a006719dd711fc67" candidate_uuid: "6ce2ba1c40a248919ee589726ec8e561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff703e4a46b4360af610690bf372514" is_pre_election: true
I20260929 15:57:28.074146 26339 raft_consensus.cc:2454] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce2ba1c40a248919ee589726ec8e561 in term 0.
I20260929 15:57:28.074378 26548 leader_election.cc:302] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce2ba1c40a248919ee589726ec8e561, dbc0949aa70b485096df5ce8f17dbdb2; no voters:
I20260929 15:57:28.074807 26692 raft_consensus.cc:2781] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:28.074971 26692 raft_consensus.cc:482] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:28.075088 26692 raft_consensus.cc:3037] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:28.077289 26692 raft_consensus.cc:504] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.077901 26692 leader_election.cc:288] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [CANDIDATE]: Term 1 election: Requested vote from peers dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849), 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807)
I20260929 15:57:28.078333 26339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ababb0962674d36a006719dd711fc67" candidate_uuid: "6ce2ba1c40a248919ee589726ec8e561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff703e4a46b4360af610690bf372514"
I20260929 15:57:28.078327 26478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ababb0962674d36a006719dd711fc67" candidate_uuid: "6ce2ba1c40a248919ee589726ec8e561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc0949aa70b485096df5ce8f17dbdb2"
I20260929 15:57:28.078537 26478 raft_consensus.cc:3037] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:28.078537 26339 raft_consensus.cc:3037] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:28.081816 26339 raft_consensus.cc:2454] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce2ba1c40a248919ee589726ec8e561 in term 1.
I20260929 15:57:28.081816 26478 raft_consensus.cc:2454] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce2ba1c40a248919ee589726ec8e561 in term 1.
I20260929 15:57:28.082242 26551 leader_election.cc:302] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce2ba1c40a248919ee589726ec8e561, 7ff703e4a46b4360af610690bf372514; no voters:
I20260929 15:57:28.082508 26692 raft_consensus.cc:2781] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:28.082965 26692 raft_consensus.cc:686] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 LEADER]: Becoming Leader. State: Replica: 6ce2ba1c40a248919ee589726ec8e561, State: Running, Role: LEADER
I20260929 15:57:28.083546 26692 consensus_queue.cc:237] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.088012 26181 catalog_manager.cc:5495] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67). New cstate: current_term: 1 leader_uuid: "6ce2ba1c40a248919ee589726ec8e561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } health_report { overall_health: HEALTHY } } }
W20260929 15:57:28.113868 26390 tablet.cc:2369] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:57:28.125887 26529 tablet.cc:2369] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:57:28.144199 26668 tablet.cc:2369] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 15:57:28 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:28.150624 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.137772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34507 (local address 127.0.0.1:48148)
0929 15:57:28.138347 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.138363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.138489 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.138773 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.138776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.139537 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:28.139799 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.139807 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.140735 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.140739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.141684 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.141694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.142362 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.142366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.142491 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.143148 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:28.143164 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:28.145465 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:28.148542 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:28.148552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:28.148557 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:28.148921 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:28.149247 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:28.149250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:28.149252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:28.149397 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:28.149834 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:28.149839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:28.149981 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.150105 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.150289 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":319,"threads_started":1}
I20260929 15:57:28.150686 26277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.137670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.23.222.126:34507)
0929 15:57:28.137875 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.137881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.138445 (+ 564us) server_negotiation.cc:407] Connection header received
0929 15:57:28.138588 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.138591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.138651 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.138734 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:28.139931 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.140633 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.141794 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.142214 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.142300 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.145622 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:28.145626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:28.145630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:28.145657 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:28.148428 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:28.149049 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:28.149053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:28.149055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:28.149160 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:28.149481 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:28.149484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:28.149486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:28.149735 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:28.149835 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.150188 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.150413 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":63}
I20260929 15:57:28.155583 26181 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:48148:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:57:28.156582 26181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:28.163012 26319 tablet_service.cc:1467] Processing CreateTablet for tablet 196b31fde4ab44d4b65dd53cee8f462b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=544071b16c1c4eae943a31a3eb0e735d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:28.163187 26458 tablet_service.cc:1467] Processing CreateTablet for tablet 196b31fde4ab44d4b65dd53cee8f462b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=544071b16c1c4eae943a31a3eb0e735d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:28.163295 26597 tablet_service.cc:1467] Processing CreateTablet for tablet 196b31fde4ab44d4b65dd53cee8f462b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=544071b16c1c4eae943a31a3eb0e735d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:57:28.164003 26319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196b31fde4ab44d4b65dd53cee8f462b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:28.164099 26597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196b31fde4ab44d4b65dd53cee8f462b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:28.164099 26458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 196b31fde4ab44d4b65dd53cee8f462b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:28.169343 26688 tablet_bootstrap.cc:492] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: Bootstrap starting.
I20260929 15:57:28.170720 26688 tablet_bootstrap.cc:654] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:28.171458 26686 tablet_bootstrap.cc:492] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: Bootstrap starting.
I20260929 15:57:28.171195 26687 tablet_bootstrap.cc:492] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: Bootstrap starting.
I20260929 15:57:28.173174 26686 tablet_bootstrap.cc:654] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:28.173187 26688 tablet_bootstrap.cc:492] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: No bootstrap required, opened a new log
I20260929 15:57:28.173341 26688 ts_tablet_manager.cc:1397] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:57:28.173445 26687 tablet_bootstrap.cc:654] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:28.173995 26688 raft_consensus.cc:348] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.174274 26688 raft_consensus.cc:374] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:28.174377 26688 raft_consensus.cc:729] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff703e4a46b4360af610690bf372514, State: Initialized, Role: FOLLOWER
I20260929 15:57:28.174710 26688 consensus_queue.cc:260] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.175186 26694 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.175601 26694 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:28.176092 26686 tablet_bootstrap.cc:492] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: No bootstrap required, opened a new log
I20260929 15:57:28.176111 26688 ts_tablet_manager.cc:1428] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:28.176162 26687 tablet_bootstrap.cc:492] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: No bootstrap required, opened a new log
I20260929 15:57:28.176222 26686 ts_tablet_manager.cc:1397] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 15:57:28.176290 26687 ts_tablet_manager.cc:1397] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 15:57:28.176975 26686 raft_consensus.cc:348] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.177062 26687 raft_consensus.cc:348] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.177234 26686 raft_consensus.cc:374] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:28.177276 26687 raft_consensus.cc:374] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:28.177342 26686 raft_consensus.cc:729] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2ba1c40a248919ee589726ec8e561, State: Initialized, Role: FOLLOWER
I20260929 15:57:28.177378 26687 raft_consensus.cc:729] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc0949aa70b485096df5ce8f17dbdb2, State: Initialized, Role: FOLLOWER
I20260929 15:57:28.177575 26686 consensus_queue.cc:260] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.177636 26687 consensus_queue.cc:260] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.177978 26692 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.178218 26687 ts_tablet_manager.cc:1428] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:57:28.178260 26692 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:28.178731 26686 ts_tablet_manager.cc:1428] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 15:57:28.178627 26693 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.178980 26693 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:28.299450 26693 raft_consensus.cc:482] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:28.299809 26693 raft_consensus.cc:504] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.301451 26693 leader_election.cc:288] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807), 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569)
I20260929 15:57:28.310175 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.301865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:43595 (local address 127.23.222.67:38569)
0929 15:57:28.302115 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.302123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.302781 (+ 658us) server_negotiation.cc:407] Connection header received
0929 15:57:28.303034 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.303039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.303117 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.303216 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:28.303969 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.305029 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.307826 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.309045 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.309217 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.309289 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.309789 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.309920 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:57:28.310201 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.301724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:57317 (local address 127.23.222.65:36807)
0929 15:57:28.301909 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.301917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.302595 (+ 678us) server_negotiation.cc:407] Connection header received
0929 15:57:28.302830 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.302837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.302920 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.303002 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:28.303704 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.304892 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.307831 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.309066 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.309217 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.309289 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.309789 (+ 500us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.309920 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:57:28.310503 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.301673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36807 (local address 127.23.222.66:57317)
0929 15:57:28.302497 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.302519 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.302676 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.303053 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.303057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.303079 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:28.303537 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.303548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.305018 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.305022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.307671 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.307682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.309330 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.309336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.309581 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.309656 (+ 75us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.309747 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.309934 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":255,"thread_start_us":440,"threads_started":1}
I20260929 15:57:28.310539 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.301792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:38569 (local address 127.23.222.66:43595)
0929 15:57:28.302678 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.302697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.302884 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.303278 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.303283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.303311 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:28.303786 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.303797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.305162 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.305173 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.307668 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.307682 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.309323 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.309330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.309584 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.309648 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.309747 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.309934 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":246,"thread_start_us":577,"threads_started":1}
I20260929 15:57:28.311146 26339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196b31fde4ab44d4b65dd53cee8f462b" candidate_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff703e4a46b4360af610690bf372514" is_pre_election: true
I20260929 15:57:28.311451 26339 raft_consensus.cc:2454] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc0949aa70b485096df5ce8f17dbdb2 in term 0.
I20260929 15:57:28.311429 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196b31fde4ab44d4b65dd53cee8f462b" candidate_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2ba1c40a248919ee589726ec8e561" is_pre_election: true
I20260929 15:57:28.311810 26619 raft_consensus.cc:2454] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbc0949aa70b485096df5ce8f17dbdb2 in term 0.
I20260929 15:57:28.311919 26412 leader_election.cc:302] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff703e4a46b4360af610690bf372514, dbc0949aa70b485096df5ce8f17dbdb2; no voters:
I20260929 15:57:28.312330 26693 raft_consensus.cc:2781] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:28.312461 26693 raft_consensus.cc:482] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:28.312569 26693 raft_consensus.cc:3037] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:28.314790 26693 raft_consensus.cc:504] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.315421 26693 leader_election.cc:288] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [CANDIDATE]: Term 1 election: Requested vote from peers 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807), 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569)
I20260929 15:57:28.315840 26339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196b31fde4ab44d4b65dd53cee8f462b" candidate_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff703e4a46b4360af610690bf372514"
I20260929 15:57:28.315858 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "196b31fde4ab44d4b65dd53cee8f462b" candidate_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2ba1c40a248919ee589726ec8e561"
I20260929 15:57:28.316058 26619 raft_consensus.cc:3037] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:28.316080 26339 raft_consensus.cc:3037] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:28.319078 26339 raft_consensus.cc:2454] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc0949aa70b485096df5ce8f17dbdb2 in term 1.
I20260929 15:57:28.319094 26619 raft_consensus.cc:2454] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbc0949aa70b485096df5ce8f17dbdb2 in term 1.
I20260929 15:57:28.319535 26411 leader_election.cc:302] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce2ba1c40a248919ee589726ec8e561, dbc0949aa70b485096df5ce8f17dbdb2; no voters:
I20260929 15:57:28.319820 26693 raft_consensus.cc:2781] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:28.320277 26693 raft_consensus.cc:686] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 LEADER]: Becoming Leader. State: Replica: dbc0949aa70b485096df5ce8f17dbdb2, State: Running, Role: LEADER
I20260929 15:57:28.320816 26693 consensus_queue.cc:237] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.324131 26709 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: TxnStatusTablet state changed. Reason: New leader dbc0949aa70b485096df5ce8f17dbdb2. Latest consensus state: current_term: 1 leader_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.324550 26709 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:28.325470 26182 catalog_manager.cc:5495] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66). New cstate: current_term: 1 leader_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20260929 15:57:28.326782 26711 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:57:28.327051 26711 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:57:28.348192 26703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.341700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35849 (local address 127.0.0.1:58174)
0929 15:57:28.341948 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.341965 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.342089 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.342596 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.342599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.342609 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:28.342757 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.342764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.344271 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.344278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.345267 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.345278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.346349 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.346356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.346576 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.346636 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.347743 (+ 1107us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.347763 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.347836 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.347955 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 15:57:28.348356 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.341814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.23.222.66:35849)
0929 15:57:28.341996 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.342003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.342025 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:57:28.342287 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.342294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.342410 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.342565 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:28.342882 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.344125 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.345415 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.346098 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.346250 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.346830 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.347629 (+ 799us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.347780 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.347891 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.347995 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 15:57:28.356886 26619 raft_consensus.cc:1261] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Refusing update from remote peer dbc0949aa70b485096df5ce8f17dbdb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:28.356969 26339 raft_consensus.cc:1261] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Refusing update from remote peer dbc0949aa70b485096df5ce8f17dbdb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:28.357962 26709 consensus_queue.cc:1035] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:28.358347 26693 consensus_queue.cc:1035] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:28.373883 26692 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: TxnStatusTablet state changed. Reason: New leader dbc0949aa70b485096df5ce8f17dbdb2. Latest consensus state: current_term: 1 leader_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.374289 26692 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:28.377077 26714 mvcc.cc:205] Tried to move back new op lower bound from 7334696748460023808 to 7334696748325871616. Current Snapshot: MvccSnapshot[applied={T|T < 7334696748460023808}]
I20260929 15:57:28.377825 26709 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.378109 26709 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:28.389912 26693 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.389995 26694 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: TxnStatusTablet state changed. Reason: New leader dbc0949aa70b485096df5ce8f17dbdb2. Latest consensus state: current_term: 1 leader_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.390213 26693 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P dbc0949aa70b485096df5ce8f17dbdb2: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:57:28.390281 26694 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:28.392513 26692 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.392850 26692 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P 6ce2ba1c40a248919ee589726ec8e561: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:28.392872 26716 mvcc.cc:205] Tried to move back new op lower bound from 7334696748460023808 to 7334696748325871616. Current Snapshot: MvccSnapshot[applied={T|T < 7334696748460023808}]
I20260929 15:57:28.398801 26694 tablet_replica.cc:439] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } } }
I20260929 15:57:28.399045 26694 tablet_replica.cc:442] T 196b31fde4ab44d4b65dd53cee8f462b P 7ff703e4a46b4360af610690bf372514: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:57:28.411345 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.404550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.23.222.67:38569)
0929 15:57:28.404697 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.404702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.404975 (+ 273us) server_negotiation.cc:407] Connection header received
0929 15:57:28.405240 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.405246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.405349 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.405502 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:28.405897 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.407003 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.408980 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.409549 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.409643 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.410177 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.410688 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.410805 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.410929 (+ 124us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.411005 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 15:57:28.428153 26725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.422029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35849 (local address 127.23.222.67:57557)
0929 15:57:28.422763 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.422778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.422890 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.423228 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.423231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.423282 (+ 51us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:28.423550 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.423558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.425001 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.425004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.425990 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.426000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.427023 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.427027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.427140 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.427189 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.427680 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.427688 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.427754 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.427874 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":434,"threads_started":1}
I20260929 15:57:28.428229 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.422133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:57557 (local address 127.23.222.66:35849)
0929 15:57:28.422342 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.422349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.422863 (+ 514us) server_negotiation.cc:407] Connection header received
0929 15:57:28.423028 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.423034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.423118 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.423217 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:28.423702 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.424870 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.426131 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.426848 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.426985 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.427441 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.427579 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.427670 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.427835 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.427919 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:57:28.438735 26339 raft_consensus.cc:1261] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Refusing update from remote peer 6ce2ba1c40a248919ee589726ec8e561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:28.439105 26478 raft_consensus.cc:1261] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Refusing update from remote peer 6ce2ba1c40a248919ee589726ec8e561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:28.439399 26696 consensus_queue.cc:1035] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:28.439869 26692 consensus_queue.cc:1035] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:28.447896 26715 mvcc.cc:205] Tried to move back new op lower bound from 7334696748798853120 to 7334696747354263552. Current Snapshot: MvccSnapshot[applied={T|T < 7334696748798853120}]
I20260929 15:57:28.479753 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.471800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:46113 (local address 127.23.222.67:38569)
0929 15:57:28.472327 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.472335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.472728 (+ 393us) server_negotiation.cc:407] Connection header received
0929 15:57:28.472936 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.472940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.473010 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.473092 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:28.473923 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.475067 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.477336 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.478004 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.478107 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.478804 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.478938 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.479013 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.479292 (+ 279us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.479372 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364}
I20260929 15:57:28.479799 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.471843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:38569 (local address 127.23.222.66:46113)
0929 15:57:28.472612 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.472633 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.472789 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.473323 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.473329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.473345 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:28.473732 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.473745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.475277 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.475282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.477161 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.477176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.478306 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.478312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.478498 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.478577 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.479109 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.479119 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.479199 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.479378 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":392,"threads_started":1}
I20260929 15:57:28.521963 26181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:28.522289 26181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:28.526541 26181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9: Sending DeleteTablet for 3 replicas of tablet 8ababb0962674d36a006719dd711fc67
I20260929 15:57:28.527740 26458 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ababb0962674d36a006719dd711fc67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:28 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:58170
I20260929 15:57:28.527863 26319 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ababb0962674d36a006719dd711fc67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:28 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:47708
I20260929 15:57:28.529130 26597 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ababb0962674d36a006719dd711fc67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:28 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:40206
I20260929 15:57:28.537392 26181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:28.538326 26181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:28.544054 26735 tablet_replica.cc:330] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: stopping tablet replica
I20260929 15:57:28.545091 26735 raft_consensus.cc:2229] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:28.546705 26735 raft_consensus.cc:2258] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:28.546928 26457 tablet_service.cc:1467] Processing CreateTablet for tablet 10c4ef7069434a388adff62965c66892 (DEFAULT_TABLE table=test-table [id=a9ac26623da140c59ce3a3aa571486d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:28.547573 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c4ef7069434a388adff62965c66892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:28.547210 26596 tablet_service.cc:1467] Processing CreateTablet for tablet 10c4ef7069434a388adff62965c66892 (DEFAULT_TABLE table=test-table [id=a9ac26623da140c59ce3a3aa571486d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:28.548414 26596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c4ef7069434a388adff62965c66892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:28.551668 26736 tablet_replica.cc:330] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: stopping tablet replica
I20260929 15:57:28.552115 26736 raft_consensus.cc:2229] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:28.552809 26736 raft_consensus.cc:2258] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:28.553660 26734 tablet_replica.cc:330] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: stopping tablet replica
I20260929 15:57:28.554113 26734 raft_consensus.cc:2229] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:28.554277 26735 ts_tablet_manager.cc:1905] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:28.554440 26734 raft_consensus.cc:2258] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:28.554703 26318 tablet_service.cc:1467] Processing CreateTablet for tablet 10c4ef7069434a388adff62965c66892 (DEFAULT_TABLE table=test-table [id=a9ac26623da140c59ce3a3aa571486d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:28.555284 26687 tablet_bootstrap.cc:492] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: Bootstrap starting.
I20260929 15:57:28.556332 26734 ts_tablet_manager.cc:1905] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:28.556466 26686 tablet_bootstrap.cc:492] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: Bootstrap starting.
I20260929 15:57:28.556965 26687 tablet_bootstrap.cc:654] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:28.558041 26318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10c4ef7069434a388adff62965c66892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:28.558251 26686 tablet_bootstrap.cc:654] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:28.564973 26734 ts_tablet_manager.cc:1918] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:28.565171 26734 log.cc:1196] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/8ababb0962674d36a006719dd711fc67
I20260929 15:57:28.565660 26688 tablet_bootstrap.cc:492] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: Bootstrap starting.
I20260929 15:57:28.565783 26734 ts_tablet_manager.cc:1939] T 8ababb0962674d36a006719dd711fc67 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting consensus metadata
I20260929 15:57:28.566128 26735 ts_tablet_manager.cc:1918] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:28.566349 26735 log.cc:1196] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/8ababb0962674d36a006719dd711fc67
I20260929 15:57:28.567022 26735 ts_tablet_manager.cc:1939] T 8ababb0962674d36a006719dd711fc67 P 7ff703e4a46b4360af610690bf372514: Deleting consensus metadata
I20260929 15:57:28.567538 26688 tablet_bootstrap.cc:654] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:28.567927 26166 catalog_manager.cc:4841] TS dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849): tablet 8ababb0962674d36a006719dd711fc67 (table test-table [id=abc96e9eb403446d9f81fbf42aac4bb7]) successfully deleted
I20260929 15:57:28.570079 26688 tablet_bootstrap.cc:492] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: No bootstrap required, opened a new log
I20260929 15:57:28.570264 26688 ts_tablet_manager.cc:1397] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 15:57:28.570461 26169 catalog_manager.cc:4841] TS 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807): tablet 8ababb0962674d36a006719dd711fc67 (table test-table [id=abc96e9eb403446d9f81fbf42aac4bb7]) successfully deleted
I20260929 15:57:28.570919 26688 raft_consensus.cc:348] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.571147 26688 raft_consensus.cc:374] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:28.571236 26688 raft_consensus.cc:729] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff703e4a46b4360af610690bf372514, State: Initialized, Role: FOLLOWER
I20260929 15:57:28.571602 26688 consensus_queue.cc:260] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.573151 26688 ts_tablet_manager.cc:1428] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 15:57:28.576659 26686 tablet_bootstrap.cc:492] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: No bootstrap required, opened a new log
I20260929 15:57:28.576942 26686 ts_tablet_manager.cc:1397] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.005s
I20260929 15:57:28.577776 26687 tablet_bootstrap.cc:492] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: No bootstrap required, opened a new log
I20260929 15:57:28.577961 26687 ts_tablet_manager.cc:1397] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.004s
I20260929 15:57:28.577952 26686 raft_consensus.cc:348] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.578349 26686 raft_consensus.cc:374] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:28.578516 26686 raft_consensus.cc:729] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2ba1c40a248919ee589726ec8e561, State: Initialized, Role: FOLLOWER
I20260929 15:57:28.578840 26687 raft_consensus.cc:348] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.579087 26687 raft_consensus.cc:374] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:28.578902 26686 consensus_queue.cc:260] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.579176 26687 raft_consensus.cc:729] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc0949aa70b485096df5ce8f17dbdb2, State: Initialized, Role: FOLLOWER
I20260929 15:57:28.579476 26687 consensus_queue.cc:260] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.581087 26686 ts_tablet_manager.cc:1428] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260929 15:57:28.582406 26687 ts_tablet_manager.cc:1428] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260929 15:57:28.581996 26736 ts_tablet_manager.cc:1905] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:28.591537 26736 ts_tablet_manager.cc:1918] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:28.591706 26736 log.cc:1196] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/8ababb0962674d36a006719dd711fc67
I20260929 15:57:28.592207 26736 ts_tablet_manager.cc:1939] T 8ababb0962674d36a006719dd711fc67 P 6ce2ba1c40a248919ee589726ec8e561: Deleting consensus metadata
I20260929 15:57:28.594033 26168 catalog_manager.cc:4841] TS 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569): tablet 8ababb0962674d36a006719dd711fc67 (table test-table [id=abc96e9eb403446d9f81fbf42aac4bb7]) successfully deleted
I20260929 15:57:28.660303 26694 raft_consensus.cc:482] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:28.660637 26694 raft_consensus.cc:504] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.662528 26694 leader_election.cc:288] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849), 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569)
I20260929 15:57:28.682550 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.663711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:35767 (local address 127.23.222.66:35849)
0929 15:57:28.663936 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.663942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.673927 (+ 9985us) server_negotiation.cc:407] Connection header received
0929 15:57:28.674507 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.674513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.674599 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.674666 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:28.675539 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.676751 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.679728 (+ 2977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.681063 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.681218 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.681294 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.682095 (+ 801us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.682227 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:57:28.683627 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.662840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35849 (local address 127.23.222.65:35767)
0929 15:57:28.673770 (+ 10930us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.673797 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.674297 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.674885 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.674891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.674928 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:28.675306 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.675318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.677293 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.677300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.679551 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.679566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.681578 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.681584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.681891 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.681928 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.683056 (+ 1128us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.683198 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10771,"mutex_wait_us":93,"spinlock_wait_cycles":18816,"thread_start_us":11114,"threads_started":1}
I20260929 15:57:28.684437 26477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c4ef7069434a388adff62965c66892" candidate_uuid: "7ff703e4a46b4360af610690bf372514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" is_pre_election: true
I20260929 15:57:28.684741 26477 raft_consensus.cc:2454] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff703e4a46b4360af610690bf372514 in term 0.
I20260929 15:57:28.684976 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.664111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:43523 (local address 127.23.222.67:38569)
0929 15:57:28.664339 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.664345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.674957 (+ 10612us) server_negotiation.cc:407] Connection header received
0929 15:57:28.675300 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.675307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.675389 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.675456 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:28.676290 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.677667 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.682035 (+ 4368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.683602 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.683732 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.683807 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.684443 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.684597 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:57:28.685614 26269 leader_election.cc:302] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff703e4a46b4360af610690bf372514, dbc0949aa70b485096df5ce8f17dbdb2; no voters:
I20260929 15:57:28.685820 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.663152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:38569 (local address 127.23.222.65:43523)
0929 15:57:28.674827 (+ 11675us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.674847 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.675102 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.675668 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.675676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.675704 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:28.676107 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.676119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.679144 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.679150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.681817 (+ 2667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.681834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.684036 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.684043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.684241 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.684280 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.685230 (+ 950us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.685385 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11517,"mutex_wait_us":30,"thread_start_us":11163,"threads_started":1}
I20260929 15:57:28.686064 26694 raft_consensus.cc:2781] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:28.686175 26694 raft_consensus.cc:482] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:28.686257 26694 raft_consensus.cc:3037] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:28.686508 26618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c4ef7069434a388adff62965c66892" candidate_uuid: "7ff703e4a46b4360af610690bf372514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2ba1c40a248919ee589726ec8e561" is_pre_election: true
I20260929 15:57:28.686779 26618 raft_consensus.cc:2454] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff703e4a46b4360af610690bf372514 in term 0.
I20260929 15:57:28.689635 26694 raft_consensus.cc:504] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.690332 26694 leader_election.cc:288] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [CANDIDATE]: Term 1 election: Requested vote from peers dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849), 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569)
I20260929 15:57:28.690814 26477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c4ef7069434a388adff62965c66892" candidate_uuid: "7ff703e4a46b4360af610690bf372514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc0949aa70b485096df5ce8f17dbdb2"
I20260929 15:57:28.691046 26477 raft_consensus.cc:3037] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:28.691088 26618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c4ef7069434a388adff62965c66892" candidate_uuid: "7ff703e4a46b4360af610690bf372514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2ba1c40a248919ee589726ec8e561"
I20260929 15:57:28.691350 26618 raft_consensus.cc:3037] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:28.694638 26477 raft_consensus.cc:2454] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff703e4a46b4360af610690bf372514 in term 1.
I20260929 15:57:28.695186 26269 leader_election.cc:302] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff703e4a46b4360af610690bf372514, dbc0949aa70b485096df5ce8f17dbdb2; no voters:
I20260929 15:57:28.695423 26618 raft_consensus.cc:2454] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff703e4a46b4360af610690bf372514 in term 1.
I20260929 15:57:28.695633 26694 raft_consensus.cc:2781] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:28.696444 26694 raft_consensus.cc:686] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff703e4a46b4360af610690bf372514, State: Running, Role: LEADER
I20260929 15:57:28.697196 26694 consensus_queue.cc:237] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:28.701256 26182 catalog_manager.cc:5495] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ff703e4a46b4360af610690bf372514 (127.23.222.65). New cstate: current_term: 1 leader_uuid: "7ff703e4a46b4360af610690bf372514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20260929 15:57:28.735636 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.727697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.23.222.65:36807)
0929 15:57:28.727864 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.727871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.728180 (+ 309us) server_negotiation.cc:407] Connection header received
0929 15:57:28.728466 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.728472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.728578 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.728750 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:28.729209 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.730342 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.732532 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.733205 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.733352 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.733890 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.734757 (+ 867us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.734918 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.735186 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.735314 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 15:57:28.756848 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.748883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:46915 (local address 127.23.222.66:35849)
0929 15:57:28.749088 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.749095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.749779 (+ 684us) server_negotiation.cc:407] Connection header received
0929 15:57:28.750001 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.750007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.750103 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.750212 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:28.750843 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.752170 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.754260 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.755030 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.755166 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.755960 (+ 794us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.756122 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.756230 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.756416 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.756518 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:57:28.756997 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.749001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35849 (local address 127.23.222.65:46915)
0929 15:57:28.749653 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.749674 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.749836 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.750248 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.750253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.750267 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:28.750644 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.750658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.752320 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.752326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.754078 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.754111 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.755331 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.755339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.755631 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.755727 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.756255 (+ 528us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.756269 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.756389 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.756557 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":283,"threads_started":1}
I20260929 15:57:28.769920 26618 raft_consensus.cc:1261] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Refusing update from remote peer 7ff703e4a46b4360af610690bf372514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:28.770126 26477 raft_consensus.cc:1261] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Refusing update from remote peer 7ff703e4a46b4360af610690bf372514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:28.770898 26742 consensus_queue.cc:1035] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:28.771457 26694 consensus_queue.cc:1035] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:28.832588 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.820568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36807 (local address 127.23.222.66:44439)
0929 15:57:28.820908 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:28.820929 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:28.821049 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:28.821992 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:28.821998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:28.822011 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:57:28.822242 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.822255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.825520 (+ 3265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.825526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.827532 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:28.827553 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.831032 (+ 3479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.831039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:28.831289 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.831362 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.831899 (+ 537us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.831913 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 15:57:28.831995 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 15:57:28.832205 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 15:57:28.833369 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:28.819961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:44439 (local address 127.23.222.65:36807)
0929 15:57:28.821511 (+ 1550us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:28.821521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:28.821542 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:57:28.821611 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:28.821618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:28.821708 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:28.821807 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:28.823099 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.824333 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.827770 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:28.828617 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:28.828782 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:28.831593 (+ 2811us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:28.831738 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:28.831826 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:28.832937 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0929 15:57:28.833032 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1369}
W20260929 15:57:28.834631 26716 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 15:57:30.489940 26181 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:30.490237 26181 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:30.494524 26181 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9: Sending DeleteTablet for 3 replicas of tablet 10c4ef7069434a388adff62965c66892
I20260929 15:57:30.495366 26318 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c4ef7069434a388adff62965c66892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:30 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:47708
I20260929 15:57:30.495440 26457 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c4ef7069434a388adff62965c66892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:30 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:58170
I20260929 15:57:30.495916 26596 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c4ef7069434a388adff62965c66892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:30 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:40206
I20260929 15:57:30.496104 26775 tablet_replica.cc:330] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: stopping tablet replica
I20260929 15:57:30.497242 26775 raft_consensus.cc:2229] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:30.497856 26775 raft_consensus.cc:2258] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:30.498858 26775 ts_tablet_manager.cc:1905] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:30.502254 26777 tablet_replica.cc:330] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: stopping tablet replica
I20260929 15:57:30.502494 26777 raft_consensus.cc:2229] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:30.502768 26777 raft_consensus.cc:2258] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:30.503165 26776 tablet_replica.cc:330] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: stopping tablet replica
I20260929 15:57:30.503454 26776 raft_consensus.cc:2229] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:30.503734 26777 ts_tablet_manager.cc:1905] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:30.503747 26776 raft_consensus.cc:2258] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:30.504858 26776 ts_tablet_manager.cc:1905] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:30.506703 26775 ts_tablet_manager.cc:1918] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:30.506841 26775 log.cc:1196] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/10c4ef7069434a388adff62965c66892
I20260929 15:57:30.507359 26775 ts_tablet_manager.cc:1939] T 10c4ef7069434a388adff62965c66892 P 7ff703e4a46b4360af610690bf372514: Deleting consensus metadata
I20260929 15:57:30.508519 26169 catalog_manager.cc:4841] TS 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807): tablet 10c4ef7069434a388adff62965c66892 (table test-table [id=a9ac26623da140c59ce3a3aa571486d3]) successfully deleted
I20260929 15:57:30.509752 26777 ts_tablet_manager.cc:1918] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:30.509904 26777 log.cc:1196] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/10c4ef7069434a388adff62965c66892
I20260929 15:57:30.510396 26777 ts_tablet_manager.cc:1939] T 10c4ef7069434a388adff62965c66892 P 6ce2ba1c40a248919ee589726ec8e561: Deleting consensus metadata
I20260929 15:57:30.511348 26168 catalog_manager.cc:4841] TS 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569): tablet 10c4ef7069434a388adff62965c66892 (table test-table [id=a9ac26623da140c59ce3a3aa571486d3]) successfully deleted
I20260929 15:57:30.512272 26776 ts_tablet_manager.cc:1918] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:30.512455 26776 log.cc:1196] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/10c4ef7069434a388adff62965c66892
I20260929 15:57:30.512879 26776 ts_tablet_manager.cc:1939] T 10c4ef7069434a388adff62965c66892 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting consensus metadata
I20260929 15:57:30.513784 26166 catalog_manager.cc:4841] TS dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849): tablet 10c4ef7069434a388adff62965c66892 (table test-table [id=a9ac26623da140c59ce3a3aa571486d3]) successfully deleted
Sep 29 15:57:30 dist-test-slave-cq88 krb5kdc[26137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:57:30 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:30.552251 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:30.532506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.23.222.126:34507)
0929 15:57:30.533078 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:30.533083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:30.533104 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:57:30.533261 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:30.533265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:30.533325 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:30.533444 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:30.534794 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:30.535681 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:30.537021 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:30.537436 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:30.537529 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:30.546061 (+ 8532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:30.546067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:30.546070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:30.546107 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:30.549219 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:30.550006 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:30.550012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:30.550016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:30.550162 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:30.550591 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:30.550598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:30.550602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:30.550913 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:30.551173 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:30.551580 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 15:57:30.551825 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":287,"threads_started":1}
I20260929 15:57:30.555711 26181 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:30.556340 26181 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:30.562835 26318 tablet_service.cc:1467] Processing CreateTablet for tablet 4518d4aaaef146afae833e717277ca95 (DEFAULT_TABLE table=test-table [id=015afce6a71d49139558462d3132b825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:30.563090 26596 tablet_service.cc:1467] Processing CreateTablet for tablet 4518d4aaaef146afae833e717277ca95 (DEFAULT_TABLE table=test-table [id=015afce6a71d49139558462d3132b825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:30.563149 26457 tablet_service.cc:1467] Processing CreateTablet for tablet 4518d4aaaef146afae833e717277ca95 (DEFAULT_TABLE table=test-table [id=015afce6a71d49139558462d3132b825]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:30.563386 26318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4518d4aaaef146afae833e717277ca95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:30.563578 26596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4518d4aaaef146afae833e717277ca95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:30.563612 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4518d4aaaef146afae833e717277ca95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:30.569592 26786 tablet_bootstrap.cc:492] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: Bootstrap starting.
I20260929 15:57:30.570545 26787 tablet_bootstrap.cc:492] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: Bootstrap starting.
I20260929 15:57:30.571480 26788 tablet_bootstrap.cc:492] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: Bootstrap starting.
I20260929 15:57:30.571499 26786 tablet_bootstrap.cc:654] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:30.572227 26787 tablet_bootstrap.cc:654] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:30.573112 26788 tablet_bootstrap.cc:654] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:30.575095 26788 tablet_bootstrap.cc:492] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: No bootstrap required, opened a new log
I20260929 15:57:30.575304 26788 ts_tablet_manager.cc:1397] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260929 15:57:30.575555 26787 tablet_bootstrap.cc:492] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: No bootstrap required, opened a new log
I20260929 15:57:30.575718 26787 ts_tablet_manager.cc:1397] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260929 15:57:30.576176 26786 tablet_bootstrap.cc:492] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: No bootstrap required, opened a new log
I20260929 15:57:30.576332 26786 ts_tablet_manager.cc:1397] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 15:57:30.576182 26788 raft_consensus.cc:348] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.576419 26788 raft_consensus.cc:374] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:30.576514 26788 raft_consensus.cc:729] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff703e4a46b4360af610690bf372514, State: Initialized, Role: FOLLOWER
I20260929 15:57:30.576519 26787 raft_consensus.cc:348] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.576758 26787 raft_consensus.cc:374] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:30.576856 26787 raft_consensus.cc:729] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc0949aa70b485096df5ce8f17dbdb2, State: Initialized, Role: FOLLOWER
I20260929 15:57:30.576777 26788 consensus_queue.cc:260] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.577107 26787 consensus_queue.cc:260] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.577098 26786 raft_consensus.cc:348] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.577332 26786 raft_consensus.cc:374] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:30.577446 26786 raft_consensus.cc:729] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2ba1c40a248919ee589726ec8e561, State: Initialized, Role: FOLLOWER
I20260929 15:57:30.577713 26787 ts_tablet_manager.cc:1428] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 15:57:30.577668 26786 consensus_queue.cc:260] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.578552 26786 ts_tablet_manager.cc:1428] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:57:30.579835 26788 ts_tablet_manager.cc:1428] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 15:57:30.611272 26792 raft_consensus.cc:482] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:30.611552 26792 raft_consensus.cc:504] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.612249 26792 leader_election.cc:288] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807), dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849)
I20260929 15:57:30.612819 26339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4518d4aaaef146afae833e717277ca95" candidate_uuid: "6ce2ba1c40a248919ee589726ec8e561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff703e4a46b4360af610690bf372514" is_pre_election: true
I20260929 15:57:30.612819 26477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4518d4aaaef146afae833e717277ca95" candidate_uuid: "6ce2ba1c40a248919ee589726ec8e561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" is_pre_election: true
I20260929 15:57:30.613101 26477 raft_consensus.cc:2454] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce2ba1c40a248919ee589726ec8e561 in term 0.
I20260929 15:57:30.613077 26339 raft_consensus.cc:2454] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ce2ba1c40a248919ee589726ec8e561 in term 0.
I20260929 15:57:30.613567 26548 leader_election.cc:302] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce2ba1c40a248919ee589726ec8e561, dbc0949aa70b485096df5ce8f17dbdb2; no voters:
I20260929 15:57:30.613906 26792 raft_consensus.cc:2781] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:30.614049 26792 raft_consensus.cc:482] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:30.614166 26792 raft_consensus.cc:3037] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:30.616292 26792 raft_consensus.cc:504] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.616794 26792 leader_election.cc:288] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [CANDIDATE]: Term 1 election: Requested vote from peers 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807), dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849)
I20260929 15:57:30.617377 26477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4518d4aaaef146afae833e717277ca95" candidate_uuid: "6ce2ba1c40a248919ee589726ec8e561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc0949aa70b485096df5ce8f17dbdb2"
I20260929 15:57:30.617432 26339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4518d4aaaef146afae833e717277ca95" candidate_uuid: "6ce2ba1c40a248919ee589726ec8e561" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ff703e4a46b4360af610690bf372514"
I20260929 15:57:30.617628 26339 raft_consensus.cc:3037] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:30.617628 26477 raft_consensus.cc:3037] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:30.619807 26339 raft_consensus.cc:2454] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce2ba1c40a248919ee589726ec8e561 in term 1.
I20260929 15:57:30.619917 26477 raft_consensus.cc:2454] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ce2ba1c40a248919ee589726ec8e561 in term 1.
I20260929 15:57:30.620237 26551 leader_election.cc:302] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce2ba1c40a248919ee589726ec8e561, 7ff703e4a46b4360af610690bf372514; no voters:
I20260929 15:57:30.620546 26792 raft_consensus.cc:2781] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:30.620975 26792 raft_consensus.cc:686] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 LEADER]: Becoming Leader. State: Replica: 6ce2ba1c40a248919ee589726ec8e561, State: Running, Role: LEADER
I20260929 15:57:30.621258 26792 consensus_queue.cc:237] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.623935 26182 catalog_manager.cc:5495] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67). New cstate: current_term: 1 leader_uuid: "6ce2ba1c40a248919ee589726ec8e561" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } health_report { overall_health: HEALTHY } } }
I20260929 15:57:30.644873 26796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:30.639801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.23.222.67:38569)
0929 15:57:30.640381 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:30.640385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:30.640401 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:57:30.640473 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:30.640476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:30.640533 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:30.640639 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:30.641019 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:30.641759 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:30.643026 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:30.643569 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:30.643699 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:30.644126 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:30.644222 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:30.644319 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:30.644477 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 15:57:30.644551 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":295,"threads_started":1}
I20260929 15:57:30.653483 26477 raft_consensus.cc:1261] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Refusing update from remote peer 6ce2ba1c40a248919ee589726ec8e561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:30.653527 26339 raft_consensus.cc:1261] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Refusing update from remote peer 6ce2ba1c40a248919ee589726ec8e561: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:30.654147 26793 consensus_queue.cc:1035] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:30.654506 26792 consensus_queue.cc:1035] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:30.698279 26182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:30.698524 26182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:30.701376 26182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9: Sending DeleteTablet for 3 replicas of tablet 4518d4aaaef146afae833e717277ca95
I20260929 15:57:30.702159 26318 tablet_service.cc:1514] Processing DeleteTablet for tablet 4518d4aaaef146afae833e717277ca95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:30 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:47708
I20260929 15:57:30.702325 26596 tablet_service.cc:1514] Processing DeleteTablet for tablet 4518d4aaaef146afae833e717277ca95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:30 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:40206
I20260929 15:57:30.702593 26777 tablet_replica.cc:330] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: stopping tablet replica
I20260929 15:57:30.702617 26775 tablet_replica.cc:330] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: stopping tablet replica
I20260929 15:57:30.702749 26457 tablet_service.cc:1514] Processing DeleteTablet for tablet 4518d4aaaef146afae833e717277ca95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:30 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:58170
I20260929 15:57:30.702807 26775 raft_consensus.cc:2229] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:30.702807 26777 raft_consensus.cc:2229] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:30.702998 26775 raft_consensus.cc:2258] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:30.703042 26776 tablet_replica.cc:330] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: stopping tablet replica
I20260929 15:57:30.703133 26777 raft_consensus.cc:2258] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:30.703277 26776 raft_consensus.cc:2229] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:30.703487 26776 raft_consensus.cc:2258] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:30.703789 26182 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 15:57:30.704135 26775 ts_tablet_manager.cc:1905] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:30.704135 26777 ts_tablet_manager.cc:1905] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 15:57:30.704407 26182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:30.707082 26776 ts_tablet_manager.cc:1905] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:30.710659 26777 ts_tablet_manager.cc:1918] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:30.710819 26777 log.cc:1196] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/4518d4aaaef146afae833e717277ca95
I20260929 15:57:30.711288 26777 ts_tablet_manager.cc:1939] T 4518d4aaaef146afae833e717277ca95 P 6ce2ba1c40a248919ee589726ec8e561: Deleting consensus metadata
I20260929 15:57:30.711761 26318 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd62a219ff2490ba7258068c94db1c9 (DEFAULT_TABLE table=test-table [id=1cb1e15c213a431a8c5172bbac58160b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:30.711882 26596 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd62a219ff2490ba7258068c94db1c9 (DEFAULT_TABLE table=test-table [id=1cb1e15c213a431a8c5172bbac58160b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:30.712160 26318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd62a219ff2490ba7258068c94db1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:30.712251 26596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd62a219ff2490ba7258068c94db1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:30.713361 26776 ts_tablet_manager.cc:1918] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:30.713502 26776 log.cc:1196] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/4518d4aaaef146afae833e717277ca95
I20260929 15:57:30.713884 26776 ts_tablet_manager.cc:1939] T 4518d4aaaef146afae833e717277ca95 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting consensus metadata
I20260929 15:57:30.715204 26168 catalog_manager.cc:4841] TS 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569): tablet 4518d4aaaef146afae833e717277ca95 (table test-table [id=015afce6a71d49139558462d3132b825]) successfully deleted
I20260929 15:57:30.715420 26166 catalog_manager.cc:4841] TS dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849): tablet 4518d4aaaef146afae833e717277ca95 (table test-table [id=015afce6a71d49139558462d3132b825]) successfully deleted
I20260929 15:57:30.715917 26457 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd62a219ff2490ba7258068c94db1c9 (DEFAULT_TABLE table=test-table [id=1cb1e15c213a431a8c5172bbac58160b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:30.716300 26457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd62a219ff2490ba7258068c94db1c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:30.717684 26786 tablet_bootstrap.cc:492] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: Bootstrap starting.
I20260929 15:57:30.718959 26788 tablet_bootstrap.cc:492] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: Bootstrap starting.
I20260929 15:57:30.719609 26786 tablet_bootstrap.cc:654] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:30.720615 26788 tablet_bootstrap.cc:654] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:30.721289 26775 ts_tablet_manager.cc:1918] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:57:30.721477 26775 log.cc:1196] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/4518d4aaaef146afae833e717277ca95
I20260929 15:57:30.721982 26775 ts_tablet_manager.cc:1939] T 4518d4aaaef146afae833e717277ca95 P 7ff703e4a46b4360af610690bf372514: Deleting consensus metadata
I20260929 15:57:30.722890 26787 tablet_bootstrap.cc:492] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: Bootstrap starting.
I20260929 15:57:30.723328 26169 catalog_manager.cc:4841] TS 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807): tablet 4518d4aaaef146afae833e717277ca95 (table test-table [id=015afce6a71d49139558462d3132b825]) successfully deleted
I20260929 15:57:30.724750 26788 tablet_bootstrap.cc:492] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: No bootstrap required, opened a new log
I20260929 15:57:30.724861 26788 ts_tablet_manager.cc:1397] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260929 15:57:30.724972 26787 tablet_bootstrap.cc:654] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:30.725455 26788 raft_consensus.cc:348] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.725661 26788 raft_consensus.cc:374] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:30.725709 26788 raft_consensus.cc:729] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff703e4a46b4360af610690bf372514, State: Initialized, Role: FOLLOWER
I20260929 15:57:30.725946 26788 consensus_queue.cc:260] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.727283 26788 ts_tablet_manager.cc:1428] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 15:57:30.728379 26787 tablet_bootstrap.cc:492] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: No bootstrap required, opened a new log
I20260929 15:57:30.728544 26787 ts_tablet_manager.cc:1397] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 15:57:30.729282 26787 raft_consensus.cc:348] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.729503 26787 raft_consensus.cc:374] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:30.729586 26787 raft_consensus.cc:729] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbc0949aa70b485096df5ce8f17dbdb2, State: Initialized, Role: FOLLOWER
I20260929 15:57:30.729815 26787 consensus_queue.cc:260] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.730370 26787 ts_tablet_manager.cc:1428] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 15:57:30.732875 26786 tablet_bootstrap.cc:492] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: No bootstrap required, opened a new log
I20260929 15:57:30.733111 26786 ts_tablet_manager.cc:1397] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260929 15:57:30.733762 26786 raft_consensus.cc:348] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.734048 26786 raft_consensus.cc:374] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:30.734157 26786 raft_consensus.cc:729] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce2ba1c40a248919ee589726ec8e561, State: Initialized, Role: FOLLOWER
I20260929 15:57:30.734426 26786 consensus_queue.cc:260] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:30.735023 26786 ts_tablet_manager.cc:1428] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 15:57:31.041105 26742 raft_consensus.cc:482] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:31.041376 26742 raft_consensus.cc:504] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:31.042044 26742 leader_election.cc:288] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849), 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569)
I20260929 15:57:31.042693 26477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd62a219ff2490ba7258068c94db1c9" candidate_uuid: "7ff703e4a46b4360af610690bf372514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" is_pre_election: true
I20260929 15:57:31.042765 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd62a219ff2490ba7258068c94db1c9" candidate_uuid: "7ff703e4a46b4360af610690bf372514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2ba1c40a248919ee589726ec8e561" is_pre_election: true
I20260929 15:57:31.042922 26477 raft_consensus.cc:2454] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff703e4a46b4360af610690bf372514 in term 0.
I20260929 15:57:31.043071 26619 raft_consensus.cc:2454] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ff703e4a46b4360af610690bf372514 in term 0.
I20260929 15:57:31.043437 26269 leader_election.cc:302] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff703e4a46b4360af610690bf372514, dbc0949aa70b485096df5ce8f17dbdb2; no voters:
I20260929 15:57:31.043748 26742 raft_consensus.cc:2781] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:31.043874 26742 raft_consensus.cc:482] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:31.044016 26742 raft_consensus.cc:3037] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:31.046002 26742 raft_consensus.cc:504] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:31.046705 26742 leader_election.cc:288] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [CANDIDATE]: Term 1 election: Requested vote from peers dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849), 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569)
I20260929 15:57:31.047072 26477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd62a219ff2490ba7258068c94db1c9" candidate_uuid: "7ff703e4a46b4360af610690bf372514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbc0949aa70b485096df5ce8f17dbdb2"
I20260929 15:57:31.047323 26477 raft_consensus.cc:3037] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:31.047230 26619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd62a219ff2490ba7258068c94db1c9" candidate_uuid: "7ff703e4a46b4360af610690bf372514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce2ba1c40a248919ee589726ec8e561"
I20260929 15:57:31.047490 26619 raft_consensus.cc:3037] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:31.049460 26477 raft_consensus.cc:2454] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff703e4a46b4360af610690bf372514 in term 1.
I20260929 15:57:31.049686 26619 raft_consensus.cc:2454] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ff703e4a46b4360af610690bf372514 in term 1.
I20260929 15:57:31.049845 26269 leader_election.cc:302] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ff703e4a46b4360af610690bf372514, dbc0949aa70b485096df5ce8f17dbdb2; no voters:
I20260929 15:57:31.050150 26742 raft_consensus.cc:2781] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:31.050549 26742 raft_consensus.cc:686] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff703e4a46b4360af610690bf372514, State: Running, Role: LEADER
I20260929 15:57:31.050806 26742 consensus_queue.cc:237] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } }
I20260929 15:57:31.053646 26182 catalog_manager.cc:5495] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ff703e4a46b4360af610690bf372514 (127.23.222.65). New cstate: current_term: 1 leader_uuid: "7ff703e4a46b4360af610690bf372514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ff703e4a46b4360af610690bf372514" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 } health_report { overall_health: UNKNOWN } } }
I20260929 15:57:31.106801 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:31.101630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.23.222.65:36807)
0929 15:57:31.102152 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:31.102156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:31.102175 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:57:31.102220 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:31.102223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:31.102279 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:31.102355 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:31.102783 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:31.103688 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:31.105012 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:31.105464 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:31.105569 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:31.105977 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:31.106088 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:31.106166 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:31.106294 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 15:57:31.106416 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":292,"threads_started":1}
I20260929 15:57:31.116039 26619 raft_consensus.cc:1261] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Refusing update from remote peer 7ff703e4a46b4360af610690bf372514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:31.116199 26477 raft_consensus.cc:1261] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Refusing update from remote peer 7ff703e4a46b4360af610690bf372514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:31.116650 26804 consensus_queue.cc:1035] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce2ba1c40a248919ee589726ec8e561" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:31.116995 26742 consensus_queue.cc:1035] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbc0949aa70b485096df5ce8f17dbdb2" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 15:57:31.137154 26757 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 15:57:32.732915 26182 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:32.733222 26182 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:32.738479 26182 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 519f3739025a4ba9aafac43aa730d0f9: Sending DeleteTablet for 3 replicas of tablet 6fd62a219ff2490ba7258068c94db1c9
I20260929 15:57:32.739378 26457 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd62a219ff2490ba7258068c94db1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:32 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:58170
I20260929 15:57:32.739490 26596 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd62a219ff2490ba7258068c94db1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:32 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:40206
I20260929 15:57:32.739881 26318 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd62a219ff2490ba7258068c94db1c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:32 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:47708
I20260929 15:57:32.740078 26835 tablet_replica.cc:330] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: stopping tablet replica
I20260929 15:57:32.740129 26834 tablet_replica.cc:330] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: stopping tablet replica
I20260929 15:57:32.740279 26835 raft_consensus.cc:2229] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:32.740556 26835 raft_consensus.cc:2258] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:32.740538 26834 raft_consensus.cc:2229] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:32.740926 26834 raft_consensus.cc:2258] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:32.742053 26835 ts_tablet_manager.cc:1905] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:32.742102 26834 ts_tablet_manager.cc:1905] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:32.742620 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26253
I20260929 15:57:32.750651 26835 ts_tablet_manager.cc:1918] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:32.750844 26835 log.cc:1196] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/6fd62a219ff2490ba7258068c94db1c9
I20260929 15:57:32.750751 26834 ts_tablet_manager.cc:1918] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:32.751142 26834 log.cc:1196] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/6fd62a219ff2490ba7258068c94db1c9
I20260929 15:57:32.751430 26835 ts_tablet_manager.cc:1939] T 6fd62a219ff2490ba7258068c94db1c9 P 6ce2ba1c40a248919ee589726ec8e561: Deleting consensus metadata
I20260929 15:57:32.751821 26834 ts_tablet_manager.cc:1939] T 6fd62a219ff2490ba7258068c94db1c9 P dbc0949aa70b485096df5ce8f17dbdb2: Deleting consensus metadata
I20260929 15:57:32.753100 26168 catalog_manager.cc:4841] TS 6ce2ba1c40a248919ee589726ec8e561 (127.23.222.67:38569): tablet 6fd62a219ff2490ba7258068c94db1c9 (table test-table [id=1cb1e15c213a431a8c5172bbac58160b]) successfully deleted
I20260929 15:57:32.753165 26166 catalog_manager.cc:4841] TS dbc0949aa70b485096df5ce8f17dbdb2 (127.23.222.66:35849): tablet 6fd62a219ff2490ba7258068c94db1c9 (table test-table [id=1cb1e15c213a431a8c5172bbac58160b]) successfully deleted
I20260929 15:57:32.755807 26836 tablet_replica.cc:330] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: stopping tablet replica
I20260929 15:57:32.756034 26836 raft_consensus.cc:2229] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:32.756383 26836 raft_consensus.cc:2258] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:32.757298 26836 ts_tablet_manager.cc:1905] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:32.762419 26836 ts_tablet_manager.cc:1918] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 15:57:32.762538 26836 log.cc:1196] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/6fd62a219ff2490ba7258068c94db1c9
I20260929 15:57:32.762966 26836 ts_tablet_manager.cc:1939] T 6fd62a219ff2490ba7258068c94db1c9 P 7ff703e4a46b4360af610690bf372514: Deleting consensus metadata
I20260929 15:57:32.763959 26169 catalog_manager.cc:4841] TS 7ff703e4a46b4360af610690bf372514 (127.23.222.65:36807): tablet 6fd62a219ff2490ba7258068c94db1c9 (table test-table [id=1cb1e15c213a431a8c5172bbac58160b]) successfully deleted
Sep 29 15:57:32 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:32.770718 26838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:32.743768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.23.222.65:36807)
0929 15:57:32.755860 (+ 12092us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:32.755891 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:32.755919 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:57:32.755987 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:32.755990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:32.756063 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:32.756195 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:32.757874 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.758651 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.760117 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.760693 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.760862 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:32.765280 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:32.765295 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:32.765311 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:32.765343 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:32.768063 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.768706 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.768713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.768716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.768840 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.769226 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.769235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.769237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.769552 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:32.769842 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:32.770150 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 15:57:32.770377 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11973,"thread_start_us":11816,"threads_started":1}
W20260929 15:57:32.771454 26384 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33196
W20260929 15:57:32.772100 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26253: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:32.772326 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 26253
I20260929 15:57:32.795439 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26393
Sep 29 15:57:32 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:57:32.810989 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:32.796207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47622 (local address 127.23.222.66:35849)
0929 15:57:32.796738 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:32.796743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:32.796759 (+ 16us) server_negotiation.cc:407] Connection header received
0929 15:57:32.796811 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:32.796815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:32.796872 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:32.796970 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:32.798458 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.799171 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.800497 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.800998 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.801102 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:32.805671 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:32.805686 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:32.805703 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:32.805740 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:32.808636 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.809242 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.809249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.809251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.809411 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.809723 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.809726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.809728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.810059 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:32.810300 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:32.810525 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 15:57:32.810689 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":297,"threads_started":1}
W20260929 15:57:32.811739 26523 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47622
W20260929 15:57:32.812410 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26393: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:32.812549 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 26393
I20260929 15:57:32.835083 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26532
Sep 29 15:57:32 dist-test-slave-cq88 krb5kdc[26137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:57:32.850925 26840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:32.835955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.23.222.67:38569)
0929 15:57:32.836520 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:32.836525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:32.836544 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:57:32.836603 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:32.836607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:32.836673 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:32.836783 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:32.838114 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.838869 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.840189 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:32.840601 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:32.840698 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:32.845173 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:32.845189 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:32.845205 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:32.845254 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:32.848091 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.848798 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.848807 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.848810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.848946 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:32.849362 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:32.849365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:32.849367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:32.849738 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:32.850042 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:32.850354 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 15:57:32.850577 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":293,"threads_started":1}
W20260929 15:57:32.851656 26662 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40460
W20260929 15:57:32.852252 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26532: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:32.852459 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 26532
I20260929 15:57:32.874920 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26150
I20260929 15:57:32.876186 26222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:32.992161 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 26150
2026-09-29T15:57:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8461 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:33 dist-test-slave-cq88 krb5kdc[26844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:33 dist-test-slave-cq88 krb5kdc[26844](info): set up 2 sockets
Sep 29 15:57:33 dist-test-slave-cq88 krb5kdc[26844](info): commencing operation
krb5kdc: starting...
W20260929 15:57:34.086534 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:57:34 dist-test-slave-cq88 krb5kdc[26844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:57:34.283653 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44457
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:44457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:34.603425 26857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:34.603711 26857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:34.603873 26857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:34.611074 26857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:34.611173 26857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:34.611294 26857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:34.611424 26857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:34.611478 26857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:34.621731 26857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:44457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:34.623104 26857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:34.624485 26857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:34.636881 26862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:34.636957 26863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:34.636958 26865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:34.637281 26857 server_base.cc:1034] running on GCE node
I20260929 15:57:34.638337 26857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:34.639887 26857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:34.641124 26857 hybrid_clock.cc:648] HybridClock initialized: now 1790697454641080 us; error 34 us; skew 500 ppm
I20260929 15:57:34.641481 26857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:34 dist-test-slave-cq88 krb5kdc[26844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697454, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:34.644835 26857 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:57:34.646479 26857 webserver.cc:460] Webserver started at http://127.23.222.126:42817/ using document root <none> and password file <none>
I20260929 15:57:34.647311 26857 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:34.647472 26857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:34.647830 26857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:34.651015 26857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "94a5960489cd49649e4ededed2add4ad"
format_stamp: "Formatted at 2026-09-29 15:57:34 on dist-test-slave-cq88"
server_key: "297b7988f24e829812d29acb8aacc43e"
server_key_iv: "c6a4a1565ea925f7d706feb4a7bd9f95"
server_key_version: "encryptionkey@0"
I20260929 15:57:34.651922 26857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "94a5960489cd49649e4ededed2add4ad"
format_stamp: "Formatted at 2026-09-29 15:57:34 on dist-test-slave-cq88"
server_key: "297b7988f24e829812d29acb8aacc43e"
server_key_iv: "c6a4a1565ea925f7d706feb4a7bd9f95"
server_key_version: "encryptionkey@0"
I20260929 15:57:34.658064 26857 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:57:34.661597 26872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:34.662887 26857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:57:34.663076 26857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "94a5960489cd49649e4ededed2add4ad"
format_stamp: "Formatted at 2026-09-29 15:57:34 on dist-test-slave-cq88"
server_key: "297b7988f24e829812d29acb8aacc43e"
server_key_iv: "c6a4a1565ea925f7d706feb4a7bd9f95"
server_key_version: "encryptionkey@0"
I20260929 15:57:34.663298 26857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:34.678920 26857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:34.683400 26857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:34.683704 26857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:34.702979 26857 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:44457
I20260929 15:57:34.703003 26923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:44457 every 8 connection(s)
I20260929 15:57:34.704470 26857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:34.708372 26924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:34.711474 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 26857
I20260929 15:57:34.711721 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:34.712517 24441 external_mini_cluster.cc:1428] Setting key 035153a2d864a8b238f8b0e1a086ee14
I20260929 15:57:34.722107 26924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad: Bootstrap starting.
I20260929 15:57:34.726073 26924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:34.727514 26924 log.cc:824] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:57:34 dist-test-slave-cq88 krb5kdc[26844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:34.730612 26924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad: No bootstrap required, opened a new log
I20260929 15:57:34.736908 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:34.714968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.23.222.126:44457)
0929 15:57:34.716233 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:34.716246 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:34.716286 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:57:34.717901 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:34.717927 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:34.718353 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:34.718696 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:34.720505 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.721864 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.723675 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:34.724472 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:34.724622 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:34.729607 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:34.729614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:34.729628 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:34.729661 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:34.732774 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.733606 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.733616 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.733622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.733786 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:34.734183 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:34.734186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:34.734191 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:34.734692 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:34.734893 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:34.735438 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 15:57:34.735889 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1044,"thread_start_us":785,"threads_started":1}
I20260929 15:57:34.739880 26924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a5960489cd49649e4ededed2add4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44457 } }
I20260929 15:57:34.740334 26924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:34.740446 26924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94a5960489cd49649e4ededed2add4ad, State: Initialized, Role: FOLLOWER
I20260929 15:57:34.740984 26924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a5960489cd49649e4ededed2add4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44457 } }
I20260929 15:57:34.741225 26924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:34.741375 26924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:34.741541 26924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:34.744510 26924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a5960489cd49649e4ededed2add4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44457 } }
I20260929 15:57:34.745138 26924 leader_election.cc:302] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94a5960489cd49649e4ededed2add4ad; no voters:
I20260929 15:57:34.745857 26924 leader_election.cc:288] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:34.745954 26929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:34.746649 26929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [term 1 LEADER]: Becoming Leader. State: Replica: 94a5960489cd49649e4ededed2add4ad, State: Running, Role: LEADER
I20260929 15:57:34.747133 26929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a5960489cd49649e4ededed2add4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44457 } }
I20260929 15:57:34.747913 26924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:34.750301 26931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94a5960489cd49649e4ededed2add4ad. Latest consensus state: current_term: 1 leader_uuid: "94a5960489cd49649e4ededed2add4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a5960489cd49649e4ededed2add4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44457 } } }
I20260929 15:57:34.750777 26931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:34.750332 26930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94a5960489cd49649e4ededed2add4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94a5960489cd49649e4ededed2add4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44457 } } }
I20260929 15:57:34.751310 26930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:34.756007 26934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:34.762248 26934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:34.771438 26934 catalog_manager.cc:1347] Generated new cluster ID: 2e040432f6394bd6b1343a90757f4e63
I20260929 15:57:34.771562 26934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:34.782477 26934 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:34.783847 26934 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:34.790406 26934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94a5960489cd49649e4ededed2add4ad: Generated new TSK 0
I20260929 15:57:34.791179 26934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:57:34.839898 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:44457
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:35.170032 26950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:35.170315 26950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:35.170416 26950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:35.177646 26950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:35.177866 26950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:35.178011 26950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:57:35.187904 26950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:35.189162 26950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:35.190443 26950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:35.203735 26955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:35.203846 26958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:35.203789 26956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:35.204855 26950 server_base.cc:1034] running on GCE node
I20260929 15:57:35.205571 26950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:35.206621 26950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:35.207876 26950 hybrid_clock.cc:648] HybridClock initialized: now 1790697455207796 us; error 65 us; skew 500 ppm
I20260929 15:57:35.208305 26950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:35 dist-test-slave-cq88 krb5kdc[26844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697455, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:35.211638 26950 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:57:35.213207 26950 webserver.cc:460] Webserver started at http://127.23.222.65:38119/ using document root <none> and password file <none>
I20260929 15:57:35.213940 26950 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:35.214094 26950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:35.214429 26950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:35.217746 26950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "4c85383e4be64022aeac7046504a5cff"
format_stamp: "Formatted at 2026-09-29 15:57:35 on dist-test-slave-cq88"
server_key: "9e695de169048e9c510acd8ebdfed598"
server_key_iv: "dc4d08034b4f213c23c5631da0c7d35d"
server_key_version: "encryptionkey@0"
I20260929 15:57:35.218542 26950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "4c85383e4be64022aeac7046504a5cff"
format_stamp: "Formatted at 2026-09-29 15:57:35 on dist-test-slave-cq88"
server_key: "9e695de169048e9c510acd8ebdfed598"
server_key_iv: "dc4d08034b4f213c23c5631da0c7d35d"
server_key_version: "encryptionkey@0"
I20260929 15:57:35.223784 26950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:57:35.227182 26965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:35.228792 26950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:35.229014 26950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "4c85383e4be64022aeac7046504a5cff"
format_stamp: "Formatted at 2026-09-29 15:57:35 on dist-test-slave-cq88"
server_key: "9e695de169048e9c510acd8ebdfed598"
server_key_iv: "dc4d08034b4f213c23c5631da0c7d35d"
server_key_version: "encryptionkey@0"
I20260929 15:57:35.229257 26950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:35.249043 26950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:35.253170 26950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:35.253469 26950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:35.254423 26950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:35.255750 26950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:35.255851 26950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:35.255997 26950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:35.256069 26950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:35.289096 26950 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:33283
I20260929 15:57:35.289153 27077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:33283 every 8 connection(s)
I20260929 15:57:35.290489 26950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:35.298593 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 26950
I20260929 15:57:35.298859 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:35.299527 24441 external_mini_cluster.cc:1428] Setting key b44377cb432ea4b67b20e7a497d4ffb2
Sep 29 15:57:35 dist-test-slave-cq88 krb5kdc[26844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697455, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:35.311535 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:35.293016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:33957 (local address 127.23.222.126:44457)
0929 15:57:35.293453 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:35.293458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:35.294330 (+ 872us) server_negotiation.cc:407] Connection header received
0929 15:57:35.296166 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:35.296172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:35.296272 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:35.296391 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:35.297863 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.298589 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.300313 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.300747 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.300890 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:35.304519 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:35.304524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:35.304527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:35.304562 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:35.308021 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.308859 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.308868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.308875 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.309044 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.309476 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.309483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.309489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.309868 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:35.310039 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:35.310739 (+ 700us) server_negotiation.cc:299] Negotiation successful
0929 15:57:35.311011 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":234,"threads_started":1}
I20260929 15:57:35.312198 27081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:35.293485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:44457 (local address 127.23.222.65:33957)
0929 15:57:35.294168 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:35.294212 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:35.295902 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:35.296628 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:35.296637 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:35.297149 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:35.297698 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:35.297715 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.298756 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.298766 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:35.300170 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:35.300182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.300998 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.301007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:35.301247 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:35.301912 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:35.301940 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:35.304280 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:35.308202 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:35.308210 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:35.308225 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:35.308706 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:35.309154 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:35.309158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:35.309160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:35.309344 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:35.309999 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:35.310006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:35.310352 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 15:57:35.310579 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 15:57:35.310988 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":259,"threads_started":1}
I20260929 15:57:35.313843 27078 heartbeater.cc:347] Connected to a master server at 127.23.222.126:44457
I20260929 15:57:35.314198 27078 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:35.315146 27078 heartbeater.cc:511] Master 127.23.222.126:44457 requested a full tablet report, sending...
I20260929 15:57:35.317441 26889 ts_manager.cc:194] Registered new tserver with Master: 4c85383e4be64022aeac7046504a5cff (127.23.222.65:33283)
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 15:57:35.318974 26889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:33957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:57:35.343050 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:44457
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:35.667646 27084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:35.668004 27084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:35.668203 27084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:35.675551 27084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:35.675669 27084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:35.675875 27084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:57:35.686059 27084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:35.687449 27084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:35.688791 27084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:35.702364 27092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:35.702356 27090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:35.702888 27089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:35.703451 27084 server_base.cc:1034] running on GCE node
I20260929 15:57:35.704191 27084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:35.705407 27084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:35.706651 27084 hybrid_clock.cc:648] HybridClock initialized: now 1790697455706598 us; error 48 us; skew 500 ppm
I20260929 15:57:35.707026 27084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:35 dist-test-slave-cq88 krb5kdc[26844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697455, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:35.714323 27084 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:57:35.716058 27084 webserver.cc:460] Webserver started at http://127.23.222.66:39447/ using document root <none> and password file <none>
I20260929 15:57:35.716786 27084 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:35.716931 27084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:35.717262 27084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:35.720296 27084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "66f0453147474d2d852c64cd7b7d86a8"
format_stamp: "Formatted at 2026-09-29 15:57:35 on dist-test-slave-cq88"
server_key: "74177e4c45802839617bf3098458bc98"
server_key_iv: "d65e8b996cf9eac2ab90c5d3b0742354"
server_key_version: "encryptionkey@0"
I20260929 15:57:35.721077 27084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "66f0453147474d2d852c64cd7b7d86a8"
format_stamp: "Formatted at 2026-09-29 15:57:35 on dist-test-slave-cq88"
server_key: "74177e4c45802839617bf3098458bc98"
server_key_iv: "d65e8b996cf9eac2ab90c5d3b0742354"
server_key_version: "encryptionkey@0"
I20260929 15:57:35.726223 27084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:35.729907 27099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:35.731387 27084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:57:35.731571 27084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "66f0453147474d2d852c64cd7b7d86a8"
format_stamp: "Formatted at 2026-09-29 15:57:35 on dist-test-slave-cq88"
server_key: "74177e4c45802839617bf3098458bc98"
server_key_iv: "d65e8b996cf9eac2ab90c5d3b0742354"
server_key_version: "encryptionkey@0"
I20260929 15:57:35.731796 27084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:35.751351 27084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:35.755967 27084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:35.756269 27084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:35.757292 27084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:35.758724 27084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:35.758829 27084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:35.758985 27084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:35.759058 27084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:35.795358 27084 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:44537
I20260929 15:57:35.795444 27211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:44537 every 8 connection(s)
I20260929 15:57:35.796676 27084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:35.801446 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 27084
I20260929 15:57:35.801648 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:35.802206 24441 external_mini_cluster.cc:1428] Setting key 5e3d54666faa02134b51d923ae7296b2
Sep 29 15:57:35 dist-test-slave-cq88 krb5kdc[26844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697455, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:35.819370 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:35.799350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:39773 (local address 127.23.222.126:44457)
0929 15:57:35.799562 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:35.799568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:35.800828 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 15:57:35.803352 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:35.803357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:35.803464 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:35.803574 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:35.805619 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.806300 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.808845 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.809273 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.809369 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:35.813267 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:35.813272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:35.813275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:35.813309 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:35.816120 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.817010 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.817022 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.817025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.817160 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:35.817603 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:35.817612 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:35.817614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:35.817887 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:35.818043 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:35.818745 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 15:57:35.818987 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":69}
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
I20260929 15:57:35.820410 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:35.799861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:44457 (local address 127.23.222.66:39773)
0929 15:57:35.800657 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:35.800708 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:35.803031 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:35.803798 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:35.803807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:35.804606 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:35.805441 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:35.805468 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.806474 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.806483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:35.808697 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:35.808714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:35.809516 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:35.809526 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:35.809864 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:35.810605 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:35.810627 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:35.813031 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:35.816273 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:35.816283 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:35.816341 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:35.816876 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:35.817272 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:35.817278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:35.817282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:35.817477 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:35.818017 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:35.818025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:35.818386 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 15:57:35.818652 (+ 266us) client_negotiation.cc:240] Negotiation successful
0929 15:57:35.819193 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":307,"threads_started":1}
I20260929 15:57:35.822126 27212 heartbeater.cc:347] Connected to a master server at 127.23.222.126:44457
I20260929 15:57:35.822520 27212 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
I20260929 15:57:35.823531 27212 heartbeater.cc:511] Master 127.23.222.126:44457 requested a full tablet report, sending...
I20260929 15:57:35.825332 26889 ts_manager.cc:194] Registered new tserver with Master: 66f0453147474d2d852c64cd7b7d86a8 (127.23.222.66:44537)
I20260929 15:57:35.826256 26889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:39773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:57:35.850355 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:44457
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:34063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:36.179505 27217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:36.179828 27217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:36.179999 27217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:36.189486 27217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:36.189597 27217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:36.189797 27217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:57:36.199836 27217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:34063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:44457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:36.201201 27217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:36.202553 27217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:36.215932 27222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:36.215917 27223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:36.216042 27225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:36.216826 27217 server_base.cc:1034] running on GCE node
I20260929 15:57:36.217481 27217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:36.218501 27217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:36.219753 27217 hybrid_clock.cc:648] HybridClock initialized: now 1790697456219678 us; error 52 us; skew 500 ppm
I20260929 15:57:36.220121 27217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:36 dist-test-slave-cq88 krb5kdc[26844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:36.223575 27217 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:57:36.225162 27217 webserver.cc:460] Webserver started at http://127.23.222.67:34199/ using document root <none> and password file <none>
I20260929 15:57:36.225895 27217 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:36.226037 27217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:36.226342 27217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:36.229377 27217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "351dbbef77174a238bf49c45f09f3e25"
format_stamp: "Formatted at 2026-09-29 15:57:36 on dist-test-slave-cq88"
server_key: "82ae734c0728ceba52dffd44f25c4cb5"
server_key_iv: "0582d098501b961e42599b82a19bb5a0"
server_key_version: "encryptionkey@0"
I20260929 15:57:36.230198 27217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "351dbbef77174a238bf49c45f09f3e25"
format_stamp: "Formatted at 2026-09-29 15:57:36 on dist-test-slave-cq88"
server_key: "82ae734c0728ceba52dffd44f25c4cb5"
server_key_iv: "0582d098501b961e42599b82a19bb5a0"
server_key_version: "encryptionkey@0"
I20260929 15:57:36.235826 27217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:57:36.239514 27232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:36.240868 27217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:36.241057 27217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "351dbbef77174a238bf49c45f09f3e25"
format_stamp: "Formatted at 2026-09-29 15:57:36 on dist-test-slave-cq88"
server_key: "82ae734c0728ceba52dffd44f25c4cb5"
server_key_iv: "0582d098501b961e42599b82a19bb5a0"
server_key_version: "encryptionkey@0"
I20260929 15:57:36.241253 27217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:36.269775 27217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:36.273666 27217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:36.273958 27217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:36.274933 27217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:36.276306 27217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:36.276407 27217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:36.276554 27217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:36.276624 27217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:36.308919 27217 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:42981
I20260929 15:57:36.309000 27344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:42981 every 8 connection(s)
I20260929 15:57:36.310141 27217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:36.320046 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 27217
I20260929 15:57:36.320261 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:36.320729 24441 external_mini_cluster.cc:1428] Setting key a88459662d02e49078f5d76ed876669f
I20260929 15:57:36.322299 27078 heartbeater.cc:503] Master 127.23.222.126:44457 was elected leader, sending a full tablet report...
Sep 29 15:57:36 dist-test-slave-cq88 krb5kdc[26844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697456, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:36.330662 27080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:36.312635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:39099 (local address 127.23.222.126:44457)
0929 15:57:36.312808 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:36.312813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:36.313978 (+ 1165us) server_negotiation.cc:407] Connection header received
0929 15:57:36.315907 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:36.315912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:36.316026 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:36.316133 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:36.317622 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:36.318409 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:36.320237 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:36.320714 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:36.320876 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:36.324902 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:36.324907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:36.324910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:36.324939 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:36.327536 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:36.328410 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:36.328415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:36.328417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:36.328557 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:36.329064 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:36.329067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:36.329070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:36.329362 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:36.329480 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:36.330228 (+ 748us) server_negotiation.cc:299] Negotiation successful
0929 15:57:36.330421 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:57:36.331705 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:36.313152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:44457 (local address 127.23.222.67:39099)
0929 15:57:36.313824 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:36.313863 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:36.315638 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:36.316366 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:36.316374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:36.316894 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:36.317464 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:36.317480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:36.318557 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:36.318561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:36.320051 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:36.320074 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:36.320964 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:36.320972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:36.321257 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:36.321976 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:36.322017 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:36.324723 (+ 2706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:36.327681 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:36.327694 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:36.327711 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:36.328206 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:36.328707 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:36.328711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:36.328714 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:36.328913 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:36.329500 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:36.329506 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:36.329826 (+ 320us) client_negotiation.cc:769] Sending connection context
0929 15:57:36.330084 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 15:57:36.330480 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":255,"threads_started":1}
I20260929 15:57:36.333280 27345 heartbeater.cc:347] Connected to a master server at 127.23.222.126:44457
I20260929 15:57:36.333551 27345 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:36.334300 27345 heartbeater.cc:511] Master 127.23.222.126:44457 requested a full tablet report, sending...
I20260929 15:57:36.335901 26888 ts_manager.cc:194] Registered new tserver with Master: 351dbbef77174a238bf49c45f09f3e25 (127.23.222.67:42981)
I20260929 15:57:36.336931 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:57:36.337071 26888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:39099
W20260929 15:57:36.368077 27080 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:36.365589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58098 (local address 127.23.222.126:44457)
0929 15:57:36.365833 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:36.365840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:36.366055 (+ 215us) server_negotiation.cc:407] Connection header received
0929 15:57:36.366214 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:36.366218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:36.366287 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:36.366376 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:36.367723 (+ 1347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58098: BlockingRecv error: recv got EOF from 127.0.0.1:58098 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:57:36.371037 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26950
W20260929 15:57:36.373818 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26950: Not authorized: Client connection negotiation failed: client connection to 127.23.222.65:33283: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:57:36.374008 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 26950
I20260929 15:57:36.389706 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27084
W20260929 15:57:36.392732 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27084: Not authorized: Client connection negotiation failed: client connection to 127.23.222.66:44537: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:57:36.392926 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 27084
I20260929 15:57:36.408576 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27217
W20260929 15:57:36.411441 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27217: Not authorized: Client connection negotiation failed: client connection to 127.23.222.67:42981: server requires authentication, but client does not have Kerberos credentials available
I20260929 15:57:36.411588 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 27217
I20260929 15:57:36.427031 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 26857
I20260929 15:57:36.428252 26919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:36.530169 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 26857
2026-09-29T15:57:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3535 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:36 dist-test-slave-cq88 krb5kdc[27362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:36 dist-test-slave-cq88 krb5kdc[27362](info): set up 2 sockets
Sep 29 15:57:36 dist-test-slave-cq88 krb5kdc[27362](info): commencing operation
krb5kdc: starting...
W20260929 15:57:37.634684 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:57:37 dist-test-slave-cq88 krb5kdc[27362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:57:37.831568 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:36279
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:36279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:38.154875 27375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:38.155181 27375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:38.155405 27375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:38.162743 27375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:38.162864 27375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:38.162976 27375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:38.163075 27375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:38.163164 27375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:38.173732 27375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:36279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:36279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:38.175261 27375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:38.176648 27375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:38.191802 27380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:38.191886 27383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:38.191892 27381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:38.192814 27375 server_base.cc:1034] running on GCE node
I20260929 15:57:38.193743 27375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:38.195456 27375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:38.196732 27375 hybrid_clock.cc:648] HybridClock initialized: now 1790697458196687 us; error 53 us; skew 500 ppm
I20260929 15:57:38.197139 27375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:38 dist-test-slave-cq88 krb5kdc[27362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697458, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:38.200896 27375 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:57:38.202603 27375 webserver.cc:460] Webserver started at http://127.23.222.126:32939/ using document root <none> and password file <none>
I20260929 15:57:38.203478 27375 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:38.203644 27375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:38.204010 27375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:38.207554 27375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "41597356ec334507bce59ccadf5e4236"
format_stamp: "Formatted at 2026-09-29 15:57:38 on dist-test-slave-cq88"
server_key: "4b1afb06d9b8587595d9448955e53b38"
server_key_iv: "6ed740ee06ad3875da5b9c750270212b"
server_key_version: "encryptionkey@0"
I20260929 15:57:38.208513 27375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "41597356ec334507bce59ccadf5e4236"
format_stamp: "Formatted at 2026-09-29 15:57:38 on dist-test-slave-cq88"
server_key: "4b1afb06d9b8587595d9448955e53b38"
server_key_iv: "6ed740ee06ad3875da5b9c750270212b"
server_key_version: "encryptionkey@0"
I20260929 15:57:38.214541 27375 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 15:57:38.218473 27390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:38.220016 27375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:57:38.220228 27375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "41597356ec334507bce59ccadf5e4236"
format_stamp: "Formatted at 2026-09-29 15:57:38 on dist-test-slave-cq88"
server_key: "4b1afb06d9b8587595d9448955e53b38"
server_key_iv: "6ed740ee06ad3875da5b9c750270212b"
server_key_version: "encryptionkey@0"
I20260929 15:57:38.220467 27375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:38.234858 27375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:38.239943 27375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:38.240268 27375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:38.264788 27375 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:36279
I20260929 15:57:38.264827 27441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:36279 every 8 connection(s)
I20260929 15:57:38.266417 27375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:38.270388 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 27375
I20260929 15:57:38.270629 27442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:38.270644 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:38.271538 24441 external_mini_cluster.cc:1428] Setting key 6130d12cf392725fbff36ea37fcf1112
I20260929 15:57:38.286406 27442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236: Bootstrap starting.
Sep 29 15:57:38 dist-test-slave-cq88 krb5kdc[27362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:38.289915 27442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:38.291438 27442 log.cc:824] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:38.295092 27442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236: No bootstrap required, opened a new log
I20260929 15:57:38.299139 27445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.273564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39890 (local address 127.23.222.126:36279)
0929 15:57:38.274553 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:38.274574 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:38.274628 (+ 54us) server_negotiation.cc:407] Connection header received
0929 15:57:38.276553 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:38.276590 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:38.277174 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:38.277709 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:38.279197 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.281366 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.282775 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.284061 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.284271 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.289678 (+ 5407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:38.289693 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:38.289717 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:38.289785 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:38.294036 (+ 4251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.294777 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.294787 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.294795 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.295012 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:38.295518 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:38.295523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:38.295526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:38.296362 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:38.296671 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:38.297054 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 15:57:38.297599 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":425,"threads_started":1}
I20260929 15:57:38.303584 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:38.305713 27442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41597356ec334507bce59ccadf5e4236" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36279 } }
I20260929 15:57:38.306120 27442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:38.306247 27442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41597356ec334507bce59ccadf5e4236, State: Initialized, Role: FOLLOWER
I20260929 15:57:38.306883 27442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41597356ec334507bce59ccadf5e4236" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36279 } }
I20260929 15:57:38.307129 27442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:38.307296 27442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:38.307497 27442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:38.343832 27445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:38.336369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39906 (local address 127.23.222.126:36279)
0929 15:57:38.336649 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:38.336664 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:38.337038 (+ 374us) server_negotiation.cc:407] Connection header received
0929 15:57:38.337236 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:38.337241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:38.337312 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:38.337435 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:38.338521 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.339205 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.340726 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:38.341147 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:38.341242 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:38.342314 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:38.342319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:38.342322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:38.343021 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:38.343065 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:38.343094 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:38.343180 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:38.343404 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 15:57:38.343488 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:57:38.397492 27442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41597356ec334507bce59ccadf5e4236" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36279 } }
I20260929 15:57:38.398185 27442 leader_election.cc:302] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41597356ec334507bce59ccadf5e4236; no voters:
I20260929 15:57:38.398913 27442 leader_election.cc:288] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:38.399092 27454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:38.400377 27454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [term 1 LEADER]: Becoming Leader. State: Replica: 41597356ec334507bce59ccadf5e4236, State: Running, Role: LEADER
I20260929 15:57:38.401005 27454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41597356ec334507bce59ccadf5e4236" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36279 } }
I20260929 15:57:38.401391 27442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:38.404331 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41597356ec334507bce59ccadf5e4236. Latest consensus state: current_term: 1 leader_uuid: "41597356ec334507bce59ccadf5e4236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41597356ec334507bce59ccadf5e4236" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36279 } } }
I20260929 15:57:38.404719 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:38.405297 27455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41597356ec334507bce59ccadf5e4236" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41597356ec334507bce59ccadf5e4236" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36279 } } }
I20260929 15:57:38.405663 27455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:38.407461 27460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:38.413318 27460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:38.422904 27460 catalog_manager.cc:1347] Generated new cluster ID: 1d79b26760ef4161a964c395ba19a965
I20260929 15:57:38.423012 27460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:38.439203 27460 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:38.441139 27460 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:38.456003 27460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41597356ec334507bce59ccadf5e4236: Generated new TSK 0
I20260929 15:57:38.456923 27460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:38.474635 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27375
I20260929 15:57:38.476071 27437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:38.567832 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 27375
2026-09-29T15:57:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2035 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:40 dist-test-slave-cq88 krb5kdc[27477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:40 dist-test-slave-cq88 krb5kdc[27477](info): set up 2 sockets
Sep 29 15:57:40 dist-test-slave-cq88 krb5kdc[27477](info): commencing operation
krb5kdc: starting...
W20260929 15:57:41.158526 24441 mini_kdc.cc:121] Time spent starting KDC: real 2.551s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:57:41 dist-test-slave-cq88 krb5kdc[27477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:57:41.355445 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34167
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:34167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:41.670455 27490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:41.670734 27490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:41.670823 27490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:41.678140 27490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:41.678252 27490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:41.678303 27490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:41.678367 27490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:41.678411 27490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:41.688271 27490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:34167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:41.689636 27490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:41.690958 27490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:41.703104 27498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:41.703147 27495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:41.703194 27496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:41.708904 27490 server_base.cc:1034] running on GCE node
I20260929 15:57:41.709807 27490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:41.711417 27490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:41.712642 27490 hybrid_clock.cc:648] HybridClock initialized: now 1790697461712587 us; error 67 us; skew 500 ppm
I20260929 15:57:41.712986 27490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:41 dist-test-slave-cq88 krb5kdc[27477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697461, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:41.723935 27490 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:57:41.725597 27490 webserver.cc:460] Webserver started at http://127.23.222.126:43325/ using document root <none> and password file <none>
I20260929 15:57:41.726341 27490 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:41.726481 27490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:41.726785 27490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:41.729862 27490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "1a209e6077194d48ac68ba551a6dcbac"
format_stamp: "Formatted at 2026-09-29 15:57:41 on dist-test-slave-cq88"
server_key: "b154266fc18601ed8d56d39ba58b1ba8"
server_key_iv: "84f4cdc6dd62fc35a9fcc905cf2165c7"
server_key_version: "encryptionkey@0"
I20260929 15:57:41.730611 27490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "1a209e6077194d48ac68ba551a6dcbac"
format_stamp: "Formatted at 2026-09-29 15:57:41 on dist-test-slave-cq88"
server_key: "b154266fc18601ed8d56d39ba58b1ba8"
server_key_iv: "84f4cdc6dd62fc35a9fcc905cf2165c7"
server_key_version: "encryptionkey@0"
I20260929 15:57:41.736027 27490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 15:57:41.739722 27505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:41.741101 27490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:57:41.741290 27490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "1a209e6077194d48ac68ba551a6dcbac"
format_stamp: "Formatted at 2026-09-29 15:57:41 on dist-test-slave-cq88"
server_key: "b154266fc18601ed8d56d39ba58b1ba8"
server_key_iv: "84f4cdc6dd62fc35a9fcc905cf2165c7"
server_key_version: "encryptionkey@0"
I20260929 15:57:41.741511 27490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:41.754369 27490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:41.758426 27490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:41.758718 27490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:41.777995 27556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:34167 every 8 connection(s)
I20260929 15:57:41.778007 27490 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:34167
I20260929 15:57:41.779670 27490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:41.783221 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 27490
I20260929 15:57:41.783388 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:41.783492 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:41.784116 24441 external_mini_cluster.cc:1428] Setting key 9b7e0c45ebac2bc7a77cf9b18fa13182
I20260929 15:57:41.797016 27557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac: Bootstrap starting.
Sep 29 15:57:41 dist-test-slave-cq88 krb5kdc[27477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:41.800633 27557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:41.801712 27557 log.cc:824] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:41.804716 27557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac: No bootstrap required, opened a new log
I20260929 15:57:41.807225 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:41.786065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.23.222.126:34167)
0929 15:57:41.786870 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:41.786884 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:41.786928 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:57:41.789075 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:41.789097 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:41.789491 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:41.789826 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:41.791519 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:41.792784 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:41.794616 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:41.795397 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:41.795528 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:41.800053 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:41.800061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:41.800076 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:41.800115 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:41.803166 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:41.803921 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:41.803935 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:41.803941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:41.804104 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:41.804493 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:41.804498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:41.804500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:41.805055 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:41.805389 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:41.805778 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 15:57:41.806182 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":349,"threads_started":1}
I20260929 15:57:41.812536 27557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a209e6077194d48ac68ba551a6dcbac" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34167 } }
I20260929 15:57:41.812902 27557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:41.813009 27557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a209e6077194d48ac68ba551a6dcbac, State: Initialized, Role: FOLLOWER
I20260929 15:57:41.813536 27557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a209e6077194d48ac68ba551a6dcbac" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34167 } }
I20260929 15:57:41.813766 27557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:41.813900 27557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:41.814079 27557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:41.816483 27557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a209e6077194d48ac68ba551a6dcbac" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34167 } }
I20260929 15:57:41.816946 27557 leader_election.cc:302] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a209e6077194d48ac68ba551a6dcbac; no voters:
I20260929 15:57:41.817488 27557 leader_election.cc:288] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:41.817615 27562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:41.818274 27562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [term 1 LEADER]: Becoming Leader. State: Replica: 1a209e6077194d48ac68ba551a6dcbac, State: Running, Role: LEADER
I20260929 15:57:41.818842 27562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a209e6077194d48ac68ba551a6dcbac" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34167 } }
I20260929 15:57:41.819599 27557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:41.821480 27563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a209e6077194d48ac68ba551a6dcbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a209e6077194d48ac68ba551a6dcbac" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34167 } } }
I20260929 15:57:41.822597 27563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:41.822566 27564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a209e6077194d48ac68ba551a6dcbac. Latest consensus state: current_term: 1 leader_uuid: "1a209e6077194d48ac68ba551a6dcbac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a209e6077194d48ac68ba551a6dcbac" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34167 } } }
I20260929 15:57:41.823021 27564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:41.824411 27567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:41.830400 27567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:41.839262 27567 catalog_manager.cc:1347] Generated new cluster ID: 13a69fef674840d2a5f65caedf12c085
I20260929 15:57:41.839371 27567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:41.898424 27567 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:41.899937 27567 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:41.910866 27567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a209e6077194d48ac68ba551a6dcbac: Generated new TSK 0
I20260929 15:57:41.911757 27567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:57:41.962428 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34167
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:42.298290 27583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:42.298643 27583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:42.298815 27583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:42.306450 27583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:42.306615 27583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:42.306826 27583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:57:42.316962 27583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:42.318351 27583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:42.319828 27583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:42.333201 27589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:42.333127 27591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:42.333461 27588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:42.334481 27583 server_base.cc:1034] running on GCE node
I20260929 15:57:42.335155 27583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:42.336135 27583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:42.337360 27583 hybrid_clock.cc:648] HybridClock initialized: now 1790697462337290 us; error 66 us; skew 500 ppm
I20260929 15:57:42.337688 27583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:42 dist-test-slave-cq88 krb5kdc[27477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697462, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:42.340969 27583 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:57:42.342487 27583 webserver.cc:460] Webserver started at http://127.23.222.65:35579/ using document root <none> and password file <none>
I20260929 15:57:42.343192 27583 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:42.343369 27583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:42.343683 27583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:42.346694 27583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "e25ec8bcc3fc4ccda45549aefeb0d6aa"
format_stamp: "Formatted at 2026-09-29 15:57:42 on dist-test-slave-cq88"
server_key: "0de15328ea0d6645c3736f4016842856"
server_key_iv: "3c8b5377ff683485de53569947080f8c"
server_key_version: "encryptionkey@0"
I20260929 15:57:42.347560 27583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "e25ec8bcc3fc4ccda45549aefeb0d6aa"
format_stamp: "Formatted at 2026-09-29 15:57:42 on dist-test-slave-cq88"
server_key: "0de15328ea0d6645c3736f4016842856"
server_key_iv: "3c8b5377ff683485de53569947080f8c"
server_key_version: "encryptionkey@0"
I20260929 15:57:42.352674 27583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 15:57:42.356290 27598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:42.357630 27583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:57:42.357817 27583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "e25ec8bcc3fc4ccda45549aefeb0d6aa"
format_stamp: "Formatted at 2026-09-29 15:57:42 on dist-test-slave-cq88"
server_key: "0de15328ea0d6645c3736f4016842856"
server_key_iv: "3c8b5377ff683485de53569947080f8c"
server_key_version: "encryptionkey@0"
I20260929 15:57:42.358042 27583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:42.401221 27583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:42.405368 27583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:42.405689 27583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:42.406657 27583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:42.408077 27583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:42.408294 27583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:42.408468 27583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:42.408540 27583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:42.439860 27583 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:42743
I20260929 15:57:42.439946 27710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:42743 every 8 connection(s)
I20260929 15:57:42.441211 27583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:42.442312 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 27583
I20260929 15:57:42.442516 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:42.443192 24441 external_mini_cluster.cc:1428] Setting key 27cb7902c0274c6fe959456a3cae027c
Sep 29 15:57:42 dist-test-slave-cq88 krb5kdc[27477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697462, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 15:57:42.464767 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:42.444297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:47761 (local address 127.23.222.126:34167)
0929 15:57:42.446763 (+ 2466us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:42.446772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:42.446808 (+ 36us) server_negotiation.cc:407] Connection header received
0929 15:57:42.448025 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:42.448034 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:42.448126 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:42.448263 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:42.449849 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.450883 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.453480 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.453897 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.454027 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:42.458471 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:42.458477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:42.458480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:42.458515 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:42.461387 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:42.462254 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:42.462259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:42.462262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:42.462414 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:42.462863 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:42.462870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:42.462876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:42.463152 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:42.463347 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:42.464156 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 15:57:42.464397 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2320,"thread_start_us":6125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
I20260929 15:57:42.465940 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:42.445057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34167 (local address 127.23.222.65:47761)
0929 15:57:42.445911 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:42.445950 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:42.447753 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:42.448521 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:42.448530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:42.449065 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:42.449662 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:42.449680 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.451064 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.451068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:42.453239 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:42.453260 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.454176 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.454184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:42.454544 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:42.455374 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:42.455410 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:42.458202 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:42.461536 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:42.461546 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:42.461565 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:42.462102 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:42.462515 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:42.462520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:42.462524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:42.462721 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:42.463327 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:42.463334 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:42.463692 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 15:57:42.464001 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 15:57:42.464583 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":362,"threads_started":1}
I20260929 15:57:42.467796 27711 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34167
I20260929 15:57:42.468103 27711 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:42.468850 27711 heartbeater.cc:511] Master 127.23.222.126:34167 requested a full tablet report, sending...
I20260929 15:57:42.470927 27522 ts_manager.cc:194] Registered new tserver with Master: e25ec8bcc3fc4ccda45549aefeb0d6aa (127.23.222.65:42743)
I20260929 15:57:42.472398 27522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:47761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:57:42.490803 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34167
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:42.822680 27717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:42.823065 27717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:42.823253 27717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:42.830840 27717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:42.830997 27717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:42.831178 27717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:57:42.841053 27717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:42.842425 27717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:42.843780 27717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:42.856819 27722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:42.857113 27723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:42.858405 27725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:42.858659 27717 server_base.cc:1034] running on GCE node
I20260929 15:57:42.859644 27717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:42.860697 27717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:42.861934 27717 hybrid_clock.cc:648] HybridClock initialized: now 1790697462861856 us; error 69 us; skew 500 ppm
I20260929 15:57:42.862303 27717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:42 dist-test-slave-cq88 krb5kdc[27477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697462, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:42.865777 27717 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:57:42.867395 27717 webserver.cc:460] Webserver started at http://127.23.222.66:44789/ using document root <none> and password file <none>
I20260929 15:57:42.868095 27717 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:42.868259 27717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:42.868561 27717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:42.871596 27717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "36bb4cf4c0334e9ab47bd7f07a2f6f89"
format_stamp: "Formatted at 2026-09-29 15:57:42 on dist-test-slave-cq88"
server_key: "aef3d3c4429b5a85fec16a896979c3a9"
server_key_iv: "803058b5e94650d9fe55fd3db3c9b406"
server_key_version: "encryptionkey@0"
I20260929 15:57:42.872387 27717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "36bb4cf4c0334e9ab47bd7f07a2f6f89"
format_stamp: "Formatted at 2026-09-29 15:57:42 on dist-test-slave-cq88"
server_key: "aef3d3c4429b5a85fec16a896979c3a9"
server_key_iv: "803058b5e94650d9fe55fd3db3c9b406"
server_key_version: "encryptionkey@0"
I20260929 15:57:42.877830 27717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:57:42.881695 27732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:42.883417 27717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:42.883618 27717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "36bb4cf4c0334e9ab47bd7f07a2f6f89"
format_stamp: "Formatted at 2026-09-29 15:57:42 on dist-test-slave-cq88"
server_key: "aef3d3c4429b5a85fec16a896979c3a9"
server_key_iv: "803058b5e94650d9fe55fd3db3c9b406"
server_key_version: "encryptionkey@0"
I20260929 15:57:42.883841 27717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:42.900089 27717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:42.904759 27717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:42.905078 27717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:42.906097 27717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:42.907692 27717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:42.907812 27717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:42.907959 27717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:42.908028 27717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:42.941265 27717 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:42449
I20260929 15:57:42.941296 27844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:42449 every 8 connection(s)
I20260929 15:57:42.942586 27717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:42.950030 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 27717
I20260929 15:57:42.950210 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:42.950793 24441 external_mini_cluster.cc:1428] Setting key 84d9f9ee68b170afd4eb40a34353e983
Sep 29 15:57:42 dist-test-slave-cq88 krb5kdc[27477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697462, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:42.964936 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:42.945219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:35363 (local address 127.23.222.126:34167)
0929 15:57:42.945417 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:42.945421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:42.946587 (+ 1166us) server_negotiation.cc:407] Connection header received
0929 15:57:42.948476 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:42.948482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:42.948571 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:42.948688 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:42.950231 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.951311 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.953139 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.953830 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.953994 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:42.958103 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:42.958112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:42.958118 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:42.958165 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:42.961112 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:42.962215 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:42.962220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:42.962223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:42.962364 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:42.962922 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:42.962926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:42.962928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:42.963413 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:42.963541 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:42.964488 (+ 947us) server_negotiation.cc:299] Negotiation successful
0929 15:57:42.964687 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 15:57:42.966665 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:42.945660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34167 (local address 127.23.222.66:35363)
0929 15:57:42.946432 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:42.946470 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:42.948205 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:42.948934 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:42.948944 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:42.949451 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:42.950021 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:42.950039 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.951494 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.951499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:42.952999 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:42.953011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:42.954087 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:42.954093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:42.954342 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:42.955085 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:42.955111 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:42.957812 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:42.961288 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:42.961302 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:42.961324 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:42.962061 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:42.962489 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:42.962497 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:42.962503 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:42.962774 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:42.963550 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:42.963561 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:42.964070 (+ 509us) client_negotiation.cc:769] Sending connection context
0929 15:57:42.964404 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 15:57:42.965049 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":288,"threads_started":1}
I20260929 15:57:42.968169 27845 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34167
I20260929 15:57:42.968502 27845 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:42.969307 27845 heartbeater.cc:511] Master 127.23.222.126:34167 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
I20260929 15:57:42.970845 27522 ts_manager.cc:194] Registered new tserver with Master: 36bb4cf4c0334e9ab47bd7f07a2f6f89 (127.23.222.66:42449)
I20260929 15:57:42.971799 27522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:35363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:57:42.999010 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34167
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:43.333169 27850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:43.333557 27850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:43.333714 27850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:43.341823 27850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:43.341984 27850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:43.342151 27850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:57:43.352839 27850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:43.354498 27850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:43.356145 27850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:43.370553 27855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:43.370553 27856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:43.370545 27858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:43.371546 27850 server_base.cc:1034] running on GCE node
I20260929 15:57:43.372236 27850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:43.373317 27850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:43.374584 27850 hybrid_clock.cc:648] HybridClock initialized: now 1790697463374509 us; error 65 us; skew 500 ppm
I20260929 15:57:43.374977 27850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697463, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:43.378826 27850 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:57:43.380653 27850 webserver.cc:460] Webserver started at http://127.23.222.67:38269/ using document root <none> and password file <none>
I20260929 15:57:43.381335 27850 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:43.381450 27850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:43.381711 27850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:43.384882 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "4e303d656f6a4d35af098d4eaef40051"
format_stamp: "Formatted at 2026-09-29 15:57:43 on dist-test-slave-cq88"
server_key: "2a8b65624239b4ada17bf2494871f7ac"
server_key_iv: "8eec6be8a1a6b73ce05c4e4025a19c8a"
server_key_version: "encryptionkey@0"
I20260929 15:57:43.385680 27850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "4e303d656f6a4d35af098d4eaef40051"
format_stamp: "Formatted at 2026-09-29 15:57:43 on dist-test-slave-cq88"
server_key: "2a8b65624239b4ada17bf2494871f7ac"
server_key_iv: "8eec6be8a1a6b73ce05c4e4025a19c8a"
server_key_version: "encryptionkey@0"
I20260929 15:57:43.391556 27850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 15:57:43.395176 27865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:43.396835 27850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:43.396991 27850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "4e303d656f6a4d35af098d4eaef40051"
format_stamp: "Formatted at 2026-09-29 15:57:43 on dist-test-slave-cq88"
server_key: "2a8b65624239b4ada17bf2494871f7ac"
server_key_iv: "8eec6be8a1a6b73ce05c4e4025a19c8a"
server_key_version: "encryptionkey@0"
I20260929 15:57:43.397179 27850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:43.418928 27850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:43.423647 27850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:43.423923 27850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:43.424969 27850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:43.426493 27850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:43.426573 27850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:43.426658 27850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:43.426733 27850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:43.460724 27850 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:40061
I20260929 15:57:43.460772 27977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:40061 every 8 connection(s)
I20260929 15:57:43.462147 27850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:43.469699 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 27850
I20260929 15:57:43.469910 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:43.470484 24441 external_mini_cluster.cc:1428] Setting key 00a14f4868139e878b51d863625bdd86
I20260929 15:57:43.475929 27711 heartbeater.cc:503] Master 127.23.222.126:34167 was elected leader, sending a full tablet report...
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697463, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:43.483705 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:43.464992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:43941 (local address 127.23.222.126:34167)
0929 15:57:43.465204 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:43.465209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:43.466414 (+ 1205us) server_negotiation.cc:407] Connection header received
0929 15:57:43.468363 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:43.468369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:43.468469 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:43.468580 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:43.470578 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.471588 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.473768 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.474201 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.474331 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:43.478128 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:43.478133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:43.478136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:43.478167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:43.480848 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.481567 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.481570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.481573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.481714 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.482073 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.482075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.482078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.482325 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:43.482430 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:43.483196 (+ 766us) server_negotiation.cc:299] Negotiation successful
0929 15:57:43.483461 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:57:43.484895 27980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:43.465487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34167 (local address 127.23.222.67:43941)
0929 15:57:43.466211 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:43.466283 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:43.468065 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:43.468824 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:43.468833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:43.469478 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:43.470352 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:43.470378 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.471795 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.471804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:43.473591 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:43.473604 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.474417 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.474422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:43.474631 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:43.475336 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:43.475365 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:43.477926 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:43.480986 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:43.481008 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:43.481023 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:43.481467 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:43.481803 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:43.481806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:43.481808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:43.481985 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:43.482435 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:43.482441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:43.482789 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 15:57:43.483047 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 15:57:43.483531 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":292,"threads_started":1}
I20260929 15:57:43.486348 27978 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34167
I20260929 15:57:43.486654 27978 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:43.487440 27978 heartbeater.cc:511] Master 127.23.222.126:34167 requested a full tablet report, sending...
I20260929 15:57:43.488912 27521 ts_manager.cc:194] Registered new tserver with Master: 4e303d656f6a4d35af098d4eaef40051 (127.23.222.67:40061)
I20260929 15:57:43.489799 27521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:43941
I20260929 15:57:43.498759 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:43.548992 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:43.534202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.23.222.126:34167)
0929 15:57:43.534412 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:43.534416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:43.534955 (+ 539us) server_negotiation.cc:407] Connection header received
0929 15:57:43.535116 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:43.535120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:43.535198 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:43.535371 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:43.536756 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.537502 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.538804 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.539379 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.539489 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:43.543543 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:43.543550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:43.543553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:43.543598 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:43.546381 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.547116 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.547121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.547123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.547412 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.547761 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.547765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.547767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.548101 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:43.548314 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:43.548585 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 15:57:43.548763 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 15:57:43.549583 27521 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48506
I20260929 15:57:43.551450 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27583
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:43.568020 27989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:43.552607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46882 (local address 127.23.222.65:42743)
0929 15:57:43.553242 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:43.553249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:43.553275 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:57:43.553335 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:43.553342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:43.553550 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:43.553690 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:43.554834 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.555955 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.557202 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.557804 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.557918 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:43.562196 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:43.562213 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:43.562229 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:43.562278 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:43.565466 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.566205 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.566211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.566213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.566341 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.566659 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.566662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.566664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.567002 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:43.567312 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:43.567528 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 15:57:43.567744 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":339,"threads_started":1}
W20260929 15:57:43.569386 27706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46882
W20260929 15:57:43.570537 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:43.570770 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 27583
I20260929 15:57:43.586957 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27717
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:57:43.603971 27990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:43.587973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.23.222.66:42449)
0929 15:57:43.588617 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:43.588624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:43.588650 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:57:43.588716 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:43.588722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:43.588932 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:43.589068 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:43.590376 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.591551 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.592920 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.593507 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.593618 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:43.597908 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:43.597924 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:43.597942 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:43.597987 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:43.601125 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.601898 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.601905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.601908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.602070 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.602407 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.602410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.602412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.602760 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:43.602998 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:43.603343 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 15:57:43.603559 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":354,"threads_started":1}
W20260929 15:57:43.605237 27840 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52642
W20260929 15:57:43.606369 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:43.606563 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 27717
I20260929 15:57:43.622556 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27850
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697463, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:57:43.639685 27991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:43.623551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.23.222.67:40061)
0929 15:57:43.624126 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:43.624133 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:43.624162 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:57:43.624239 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:43.624245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:43.624452 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:43.624632 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:43.626117 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.627366 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.628605 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:43.629196 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:43.629345 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:43.633792 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:43.633806 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:43.633822 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:43.633862 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:43.636919 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.637618 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.637626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.637629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.637765 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:43.638183 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:43.638186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:43.638188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:43.638576 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:43.638817 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:43.639088 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 15:57:43.639397 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":263,"threads_started":1}
W20260929 15:57:43.641000 27973 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57362
W20260929 15:57:43.642170 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27850: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 15:57:43.642359 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 27850
I20260929 15:57:43.657966 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 27490
I20260929 15:57:43.659317 27552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:43.763337 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 27490
2026-09-29T15:57:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5199 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27995](info): set up 2 sockets
Sep 29 15:57:43 dist-test-slave-cq88 krb5kdc[27995](info): commencing operation
krb5kdc: starting...
W20260929 15:57:44.859566 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:57:44 dist-test-slave-cq88 krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:57:45.067782 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:45985
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:45985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:45.398598 28008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:45.398937 28008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:45.399101 28008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:45.406637 28008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:45.406790 28008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:45.406890 28008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:45.406991 28008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:45.407066 28008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:45.417431 28008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:45985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:45985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:45.418906 28008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:45.420351 28008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:45.432647 28014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:45.432859 28016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:45.432689 28013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:45.434011 28008 server_base.cc:1034] running on GCE node
I20260929 15:57:45.434880 28008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:45.436385 28008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:45.437614 28008 hybrid_clock.cc:648] HybridClock initialized: now 1790697465437577 us; error 62 us; skew 500 ppm
I20260929 15:57:45.437958 28008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:45 dist-test-slave-cq88 krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697465, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:45.441139 28008 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:57:45.442682 28008 webserver.cc:460] Webserver started at http://127.23.222.126:34753/ using document root <none> and password file <none>
I20260929 15:57:45.443451 28008 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:45.443589 28008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:45.443907 28008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:45.446978 28008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "26cff0ec13da467d8fbbaf0b65b431bb"
format_stamp: "Formatted at 2026-09-29 15:57:45 on dist-test-slave-cq88"
server_key: "137a2e8f7e3b1a64e7ef8c9236534620"
server_key_iv: "096da105b64f635002410841188ca5e8"
server_key_version: "encryptionkey@0"
I20260929 15:57:45.447854 28008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "26cff0ec13da467d8fbbaf0b65b431bb"
format_stamp: "Formatted at 2026-09-29 15:57:45 on dist-test-slave-cq88"
server_key: "137a2e8f7e3b1a64e7ef8c9236534620"
server_key_iv: "096da105b64f635002410841188ca5e8"
server_key_version: "encryptionkey@0"
I20260929 15:57:45.453082 28008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:57:45.456831 28023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:45.458307 28008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:45.458521 28008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "26cff0ec13da467d8fbbaf0b65b431bb"
format_stamp: "Formatted at 2026-09-29 15:57:45 on dist-test-slave-cq88"
server_key: "137a2e8f7e3b1a64e7ef8c9236534620"
server_key_iv: "096da105b64f635002410841188ca5e8"
server_key_version: "encryptionkey@0"
I20260929 15:57:45.458741 28008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:45.473237 28008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:45.477720 28008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:45.478060 28008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:45.498224 28008 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:45985
I20260929 15:57:45.498224 28074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:45985 every 8 connection(s)
I20260929 15:57:45.499815 28008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:45.503661 28075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:45.507081 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 28008
I20260929 15:57:45.507359 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:45.508013 24441 external_mini_cluster.cc:1428] Setting key 395004a55411304ecdc5a6b81c796c0a
I20260929 15:57:45.517026 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb: Bootstrap starting.
I20260929 15:57:45.520764 28075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:45.521790 28075 log.cc:824] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:57:45 dist-test-slave-cq88 krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:45.525054 28075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb: No bootstrap required, opened a new log
I20260929 15:57:45.532951 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:45.510009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.23.222.126:45985)
0929 15:57:45.510779 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:45.510795 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:45.510845 (+ 50us) server_negotiation.cc:407] Connection header received
0929 15:57:45.512772 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:45.512806 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:45.513347 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:45.513804 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:45.515137 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:45.517116 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:45.518388 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:45.519511 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:45.519703 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:45.523732 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:45.523742 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:45.523772 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:45.523821 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:45.527863 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:45.528973 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:45.528983 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:45.528992 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:45.529207 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:45.529564 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:45.529568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:45.529571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:45.530327 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:45.530606 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:45.530988 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 15:57:45.531500 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":302,"threads_started":1}
I20260929 15:57:45.536069 28075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cff0ec13da467d8fbbaf0b65b431bb" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45985 } }
I20260929 15:57:45.536454 28075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:45.536533 28075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26cff0ec13da467d8fbbaf0b65b431bb, State: Initialized, Role: FOLLOWER
I20260929 15:57:45.536993 28075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cff0ec13da467d8fbbaf0b65b431bb" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45985 } }
I20260929 15:57:45.537182 28075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:45.537312 28075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:45.537487 28075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:45.539917 28075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cff0ec13da467d8fbbaf0b65b431bb" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45985 } }
I20260929 15:57:45.540378 28075 leader_election.cc:302] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26cff0ec13da467d8fbbaf0b65b431bb; no voters:
I20260929 15:57:45.540911 28075 leader_election.cc:288] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:45.541077 28080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:45.541826 28080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [term 1 LEADER]: Becoming Leader. State: Replica: 26cff0ec13da467d8fbbaf0b65b431bb, State: Running, Role: LEADER
I20260929 15:57:45.542392 28080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cff0ec13da467d8fbbaf0b65b431bb" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45985 } }
I20260929 15:57:45.542994 28075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:45.545356 28081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26cff0ec13da467d8fbbaf0b65b431bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cff0ec13da467d8fbbaf0b65b431bb" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45985 } } }
I20260929 15:57:45.545764 28081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:45.546254 28082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26cff0ec13da467d8fbbaf0b65b431bb. Latest consensus state: current_term: 1 leader_uuid: "26cff0ec13da467d8fbbaf0b65b431bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26cff0ec13da467d8fbbaf0b65b431bb" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45985 } } }
I20260929 15:57:45.546787 28082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:45.547590 28088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:45.553719 28088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:45.562927 28088 catalog_manager.cc:1347] Generated new cluster ID: f8438345a81440208d65b9cd583002ee
I20260929 15:57:45.563099 28088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:45.578003 28088 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:45.579437 28088 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:45.589988 28088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26cff0ec13da467d8fbbaf0b65b431bb: Generated new TSK 0
I20260929 15:57:45.590901 28088 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:57:45.649447 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:45985
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:45.989166 28101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:45.989480 28101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:45.989575 28101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:45.996973 28101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:45.997138 28101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:45.997259 28101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:57:46.007339 28101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:45985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:46.008625 28101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:46.009927 28101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:46.023669 28106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:46.023725 28107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:46.023738 28109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:46.024444 28101 server_base.cc:1034] running on GCE node
I20260929 15:57:46.025105 28101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:46.026211 28101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:46.027446 28101 hybrid_clock.cc:648] HybridClock initialized: now 1790697466027394 us; error 52 us; skew 500 ppm
I20260929 15:57:46.027828 28101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:46 dist-test-slave-cq88 krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:46.032084 28101 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:57:46.033736 28101 webserver.cc:460] Webserver started at http://127.23.222.65:38649/ using document root <none> and password file <none>
I20260929 15:57:46.034461 28101 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:46.034569 28101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:46.034839 28101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:46.038077 28101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "e5e72f6da6d34c6a89d5fc86b36777b3"
format_stamp: "Formatted at 2026-09-29 15:57:46 on dist-test-slave-cq88"
server_key: "9849f6b4bd5def8c86183e33b8ce88d9"
server_key_iv: "487943d3d97634f85559740f3f7fbf37"
server_key_version: "encryptionkey@0"
I20260929 15:57:46.038950 28101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "e5e72f6da6d34c6a89d5fc86b36777b3"
format_stamp: "Formatted at 2026-09-29 15:57:46 on dist-test-slave-cq88"
server_key: "9849f6b4bd5def8c86183e33b8ce88d9"
server_key_iv: "487943d3d97634f85559740f3f7fbf37"
server_key_version: "encryptionkey@0"
I20260929 15:57:46.044930 28101 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 15:57:46.049046 28116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:46.050566 28101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:46.050773 28101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "e5e72f6da6d34c6a89d5fc86b36777b3"
format_stamp: "Formatted at 2026-09-29 15:57:46 on dist-test-slave-cq88"
server_key: "9849f6b4bd5def8c86183e33b8ce88d9"
server_key_iv: "487943d3d97634f85559740f3f7fbf37"
server_key_version: "encryptionkey@0"
I20260929 15:57:46.051007 28101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:46.069911 28101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:46.074661 28101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:46.074966 28101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:46.076013 28101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:46.077452 28101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:46.077553 28101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:46.077704 28101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:46.077776 28101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:46.114372 28101 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:34079
I20260929 15:57:46.114428 28228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:34079 every 8 connection(s)
I20260929 15:57:46.116039 28101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:46.119825 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 28101
I20260929 15:57:46.120013 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:46.120635 24441 external_mini_cluster.cc:1428] Setting key b263dc9e9777c5a6ac32141992e4a2f3
Sep 29 15:57:46 dist-test-slave-cq88 krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:46.139444 28231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:46.119068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:48177 (local address 127.23.222.126:45985)
0929 15:57:46.119688 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:46.119696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:46.121076 (+ 1380us) server_negotiation.cc:407] Connection header received
0929 15:57:46.123492 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:46.123497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:46.123576 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:46.123721 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:46.125250 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.126406 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.128140 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.128701 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.128804 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:46.133200 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:46.133205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:46.133208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:46.133241 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:46.136239 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:46.137045 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:46.137058 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:46.137061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:46.137184 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:46.137621 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:46.137630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:46.137632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:46.137896 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:46.138076 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:46.138800 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 15:57:46.139041 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 15:57:46.140465 28232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:46.119757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:45985 (local address 127.23.222.65:48177)
0929 15:57:46.120833 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:46.120893 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:46.123166 (+ 2273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:46.123917 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:46.123925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:46.124481 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:46.125061 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:46.125079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.126610 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.126614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:46.127989 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:46.128002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.128955 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.128965 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:46.129304 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:46.130151 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:46.130190 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:46.132943 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:46.136405 (+ 3462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:46.136415 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:46.136435 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:46.136905 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:46.137297 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:46.137303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:46.137306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:46.137496 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:46.138026 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:46.138034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:46.138416 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 15:57:46.138697 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 15:57:46.139271 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":574,"threads_started":1}
I20260929 15:57:46.142226 28229 heartbeater.cc:347] Connected to a master server at 127.23.222.126:45985
I20260929 15:57:46.142571 28229 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
I20260929 15:57:46.143522 28229 heartbeater.cc:511] Master 127.23.222.126:45985 requested a full tablet report, sending...
I20260929 15:57:46.145859 28040 ts_manager.cc:194] Registered new tserver with Master: e5e72f6da6d34c6a89d5fc86b36777b3 (127.23.222.65:34079)
I20260929 15:57:46.147447 28040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:48177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:57:46.171893 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:45985
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:46.519160 28235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:46.519537 28235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:46.519737 28235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:46.527155 28235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:46.527364 28235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:46.527561 28235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:57:46.537736 28235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:45985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:46.539070 28235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:46.540450 28235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:46.554143 28243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:46.554263 28241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:46.554241 28240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:46.554752 28235 server_base.cc:1034] running on GCE node
I20260929 15:57:46.555691 28235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:46.556689 28235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:46.557948 28235 hybrid_clock.cc:648] HybridClock initialized: now 1790697466557870 us; error 64 us; skew 500 ppm
I20260929 15:57:46.558305 28235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:46 dist-test-slave-cq88 krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:46.561658 28235 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:57:46.563566 28235 webserver.cc:460] Webserver started at http://127.23.222.66:40463/ using document root <none> and password file <none>
I20260929 15:57:46.564567 28235 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:46.564710 28235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:46.565084 28235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:46.568415 28235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "2269467f3ed74594935ef08bfd0ef6e5"
format_stamp: "Formatted at 2026-09-29 15:57:46 on dist-test-slave-cq88"
server_key: "e05366720e77bf52a8e7b1b24532b93d"
server_key_iv: "ee0e011f8b922e8437c6aebb8244e6cd"
server_key_version: "encryptionkey@0"
I20260929 15:57:46.569352 28235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "2269467f3ed74594935ef08bfd0ef6e5"
format_stamp: "Formatted at 2026-09-29 15:57:46 on dist-test-slave-cq88"
server_key: "e05366720e77bf52a8e7b1b24532b93d"
server_key_iv: "ee0e011f8b922e8437c6aebb8244e6cd"
server_key_version: "encryptionkey@0"
I20260929 15:57:46.575035 28235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:57:46.578964 28250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:46.580677 28235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:46.580904 28235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "2269467f3ed74594935ef08bfd0ef6e5"
format_stamp: "Formatted at 2026-09-29 15:57:46 on dist-test-slave-cq88"
server_key: "e05366720e77bf52a8e7b1b24532b93d"
server_key_iv: "ee0e011f8b922e8437c6aebb8244e6cd"
server_key_version: "encryptionkey@0"
I20260929 15:57:46.581153 28235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:46.594324 28235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:46.599192 28235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:46.599592 28235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:46.600669 28235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:46.602181 28235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:46.602264 28235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:46.602409 28235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:46.602468 28235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:46.639219 28235 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:41933
I20260929 15:57:46.639307 28362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:41933 every 8 connection(s)
I20260929 15:57:46.640775 28235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:46.645252 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 28235
I20260929 15:57:46.645467 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:46.646034 24441 external_mini_cluster.cc:1428] Setting key ca794c58245d957882cd9b986f189317
Sep 29 15:57:46 dist-test-slave-cq88 krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697466, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
I20260929 15:57:46.667057 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:46.643787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:60245 (local address 127.23.222.126:45985)
0929 15:57:46.644329 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:46.644336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:46.645579 (+ 1243us) server_negotiation.cc:407] Connection header received
0929 15:57:46.647870 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:46.647874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:46.647963 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:46.648086 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:46.650542 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.651581 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.654338 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.655136 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.655365 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:46.659210 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:46.659228 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:46.659267 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:46.659316 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:46.663193 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:46.664267 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:46.664276 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:46.664282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:46.664469 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:46.665007 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:46.665013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:46.665017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:46.665419 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:46.665576 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:46.666366 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 15:57:46.666641 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":248,"threads_started":1}
I20260929 15:57:46.667912 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:46.644403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:45985 (local address 127.23.222.66:60245)
0929 15:57:46.645275 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:46.645334 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:46.647588 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:46.648409 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:46.648421 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:46.649302 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:46.650269 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:46.650304 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.651768 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.651776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:46.654106 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:46.654144 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:46.655498 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:46.655505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:46.655747 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:46.656685 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:46.656710 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:46.658931 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:46.663548 (+ 4617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:46.663563 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:46.663586 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:46.664101 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:46.664649 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:46.664653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:46.664656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:46.664863 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:46.665573 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:46.665580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:46.665957 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 15:57:46.666216 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 15:57:46.666638 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":292,"threads_started":1}
I20260929 15:57:46.669727 28363 heartbeater.cc:347] Connected to a master server at 127.23.222.126:45985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
I20260929 15:57:46.670135 28363 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:46.670918 28363 heartbeater.cc:511] Master 127.23.222.126:45985 requested a full tablet report, sending...
I20260929 15:57:46.672683 28040 ts_manager.cc:194] Registered new tserver with Master: 2269467f3ed74594935ef08bfd0ef6e5 (127.23.222.66:41933)
I20260929 15:57:46.673995 28040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:60245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:57:46.699420 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:45985
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:47.040465 28369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:47.040778 28369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:47.040932 28369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:47.048519 28369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:47.048648 28369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:47.048823 28369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:57:47.059255 28369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:45985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:47.060645 28369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:47.061990 28369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:47.075438 28374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:47.075474 28375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:47.075533 28377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:47.076690 28369 server_base.cc:1034] running on GCE node
I20260929 15:57:47.077392 28369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:47.078428 28369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:47.079627 28369 hybrid_clock.cc:648] HybridClock initialized: now 1790697467079577 us; error 45 us; skew 500 ppm
I20260929 15:57:47.079960 28369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697467, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:47.089211 28369 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:57:47.090813 28369 webserver.cc:460] Webserver started at http://127.23.222.67:33059/ using document root <none> and password file <none>
I20260929 15:57:47.091617 28369 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:47.091749 28369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:47.092064 28369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:47.095077 28369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "e59d18019c034a928d68690666250f55"
format_stamp: "Formatted at 2026-09-29 15:57:47 on dist-test-slave-cq88"
server_key: "813fa76d4f452169bf860991ed0e9476"
server_key_iv: "73dce9feebc2cf06f671bdbf7578b17e"
server_key_version: "encryptionkey@0"
I20260929 15:57:47.096005 28369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "e59d18019c034a928d68690666250f55"
format_stamp: "Formatted at 2026-09-29 15:57:47 on dist-test-slave-cq88"
server_key: "813fa76d4f452169bf860991ed0e9476"
server_key_iv: "73dce9feebc2cf06f671bdbf7578b17e"
server_key_version: "encryptionkey@0"
I20260929 15:57:47.101735 28369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:57:47.105405 28384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:47.106895 28369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:57:47.107175 28369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "e59d18019c034a928d68690666250f55"
format_stamp: "Formatted at 2026-09-29 15:57:47 on dist-test-slave-cq88"
server_key: "813fa76d4f452169bf860991ed0e9476"
server_key_iv: "73dce9feebc2cf06f671bdbf7578b17e"
server_key_version: "encryptionkey@0"
I20260929 15:57:47.107450 28369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:47.135551 28369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:47.140102 28369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:47.140424 28369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:47.141456 28369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:47.142963 28369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:47.143092 28369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:47.143271 28369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:47.143343 28369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 15:57:47.151532 28229 heartbeater.cc:503] Master 127.23.222.126:45985 was elected leader, sending a full tablet report...
I20260929 15:57:47.178318 28369 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:39325
I20260929 15:57:47.178366 28496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:39325 every 8 connection(s)
I20260929 15:57:47.179752 28369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:47.189924 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 28369
I20260929 15:57:47.190148 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:47.190701 24441 external_mini_cluster.cc:1428] Setting key ab158d47656f0b4395ac23bbc724be5c
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697467, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:47.202914 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:47.182420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:37633 (local address 127.23.222.126:45985)
0929 15:57:47.182892 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:47.182897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:47.183980 (+ 1083us) server_negotiation.cc:407] Connection header received
0929 15:57:47.186038 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:47.186044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:47.186158 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:47.186304 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:47.188251 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.189044 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.191634 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.192244 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.192374 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:47.196527 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:47.196533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:47.196536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:47.196586 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:47.199360 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.200268 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.200273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.200275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.200419 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.200931 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.200935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.200937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.201230 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:47.201393 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:47.202420 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0929 15:57:47.202634 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260929 15:57:47.204026 28500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:47.183028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:45985 (local address 127.23.222.67:37633)
0929 15:57:47.183805 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:47.183842 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:47.185741 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:47.186570 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:47.186579 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:47.187144 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:47.188041 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:47.188060 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.189235 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.189239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:47.191414 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:47.191437 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.192494 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.192500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:47.192718 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:47.193562 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:47.193588 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:47.196115 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:47.199540 (+ 3425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:47.199548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:47.199565 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:47.200059 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:47.200564 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:47.200568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:47.200570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:47.200785 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:47.201481 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:47.201489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:47.201864 (+ 375us) client_negotiation.cc:769] Sending connection context
0929 15:57:47.202265 (+ 401us) client_negotiation.cc:240] Negotiation successful
0929 15:57:47.202747 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":336,"threads_started":1}
I20260929 15:57:47.206358 28497 heartbeater.cc:347] Connected to a master server at 127.23.222.126:45985
I20260929 15:57:47.206807 28497 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:47.207785 28497 heartbeater.cc:511] Master 127.23.222.126:45985 requested a full tablet report, sending...
I20260929 15:57:47.209312 28039 ts_manager.cc:194] Registered new tserver with Master: e59d18019c034a928d68690666250f55 (127.23.222.67:39325)
I20260929 15:57:47.210196 28039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:37633
I20260929 15:57:47.220381 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:47.273413 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:47.256390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.23.222.65:34079)
0929 15:57:47.257079 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:47.257086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:47.257113 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:57:47.257171 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:47.257178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:47.257399 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:47.257569 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:47.259068 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.260136 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.261426 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.262009 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.262164 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:47.266397 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:47.266412 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:47.266428 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:47.266471 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:47.270368 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.271140 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.271149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.271151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.271341 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.271751 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.271756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.271759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.272118 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:47.272427 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:47.272773 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 15:57:47.273047 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":304,"threads_started":1}
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:47.292906 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:47.278236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.23.222.126:45985)
0929 15:57:47.278477 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:47.278484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:47.278688 (+ 204us) server_negotiation.cc:407] Connection header received
0929 15:57:47.278818 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:47.278822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:47.278902 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:47.278993 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:47.280403 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.281166 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.282517 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.282980 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.283085 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:47.287544 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:47.287549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:47.287556 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:47.287596 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:47.290245 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.290919 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.290924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.290927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.291059 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.291503 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.291506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.291508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.291863 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:47.292088 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:47.292480 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 15:57:47.292687 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 15:57:47.293565 28039 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46160
I20260929 15:57:47.294796 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 28101
I20260929 15:57:47.307698 28505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:47.295834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.23.222.65:34079)
0929 15:57:47.295997 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:47.296001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:47.296226 (+ 225us) server_negotiation.cc:407] Connection header received
0929 15:57:47.296369 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:47.296373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:47.296437 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:47.296519 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:47.297680 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.298353 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.299712 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.300147 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.300253 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:47.302491 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:47.302496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:47.302499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:47.302533 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:47.305147 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.305905 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.305909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.305911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.306035 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.306471 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.306474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.306476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.306731 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:47.306911 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:47.307200 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 15:57:47.307442 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 15:57:47.308315 28224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:47.393636 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 28101
I20260929 15:57:47.410498 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 28235
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:57:47.427585 28510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:47.411599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46906 (local address 127.23.222.66:41933)
0929 15:57:47.412134 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:47.412142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:47.412168 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:57:47.412229 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:47.412236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:47.412452 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:47.412584 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:47.413905 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.414900 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.416329 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.416915 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.417029 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:47.421503 (+ 4474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:47.421516 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:47.421531 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:47.421581 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:47.424697 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.425480 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.425488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.425490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.425656 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.426087 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.426091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.426093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.426449 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:47.426696 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:47.426999 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 15:57:47.427206 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":281,"threads_started":1}
I20260929 15:57:47.428850 28358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:47.517813 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 28235
I20260929 15:57:47.534312 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 28369
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:57:47.551785 28512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:47.535463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.23.222.67:39325)
0929 15:57:47.536066 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:47.536073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:47.536101 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:57:47.536163 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:47.536170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:47.536386 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:47.536516 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:47.537957 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.539005 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.540441 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:47.541086 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:47.541204 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:47.545620 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:47.545633 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:47.545654 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:47.545696 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:47.548633 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.549490 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.549497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.549500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.549657 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:47.550093 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:47.550097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:47.550100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:47.550511 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:47.550768 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:47.551127 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 15:57:47.551500 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":314,"threads_started":1}
I20260929 15:57:47.553249 28492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:47.636808 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 28369
I20260929 15:57:47.653367 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 28008
I20260929 15:57:47.654614 28070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:47.758039 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 28008
2026-09-29T15:57:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3994 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[28517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[28517](info): set up 2 sockets
Sep 29 15:57:47 dist-test-slave-cq88 krb5kdc[28517](info): commencing operation
krb5kdc: starting...
W20260929 15:57:48.861222 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:57:49 dist-test-slave-cq88 krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:57:49.079883 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:36285
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:49.416767 28530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:49.417073 28530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:49.417263 28530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:49.424713 28530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:49.424836 28530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:49.424937 28530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:49.425040 28530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:49.425134 28530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:49.435432 28530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:49.436856 28530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:49.438196 28530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:49.450335 28535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:49.450464 28538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:49.450366 28536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:49.451797 28530 server_base.cc:1034] running on GCE node
I20260929 15:57:49.452703 28530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:49.454384 28530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:49.455684 28530 hybrid_clock.cc:648] HybridClock initialized: now 1790697469455616 us; error 50 us; skew 500 ppm
I20260929 15:57:49.456132 28530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:49 dist-test-slave-cq88 krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:49.463879 28530 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:57:49.465677 28530 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:49.465778 28530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:49.466055 28530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:49.469220 28530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "adc4d8bedcdd4a34a74392d5a659c37b"
format_stamp: "Formatted at 2026-09-29 15:57:49 on dist-test-slave-cq88"
server_key: "68305dac4047fec5ef0906342fb89a61"
server_key_iv: "628dca16c2937cf86ff62048e7d47f1a"
server_key_version: "encryptionkey@0"
I20260929 15:57:49.470033 28530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "adc4d8bedcdd4a34a74392d5a659c37b"
format_stamp: "Formatted at 2026-09-29 15:57:49 on dist-test-slave-cq88"
server_key: "68305dac4047fec5ef0906342fb89a61"
server_key_iv: "628dca16c2937cf86ff62048e7d47f1a"
server_key_version: "encryptionkey@0"
I20260929 15:57:49.475657 28530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 15:57:49.479717 28544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:49.481128 28530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:49.481285 28530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "adc4d8bedcdd4a34a74392d5a659c37b"
format_stamp: "Formatted at 2026-09-29 15:57:49 on dist-test-slave-cq88"
server_key: "68305dac4047fec5ef0906342fb89a61"
server_key_iv: "628dca16c2937cf86ff62048e7d47f1a"
server_key_version: "encryptionkey@0"
I20260929 15:57:49.481485 28530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:49.509029 28530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:49.513242 28530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:49.513489 28530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:49.532897 28530 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:36285
I20260929 15:57:49.532917 28595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:36285 every 8 connection(s)
I20260929 15:57:49.533882 28530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:49.537761 28596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:49.540146 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 28530
I20260929 15:57:49.540382 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:49.541014 24441 external_mini_cluster.cc:1428] Setting key 421a77866a6dd4efc5232c1e0592b04b
I20260929 15:57:49.550019 28596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b: Bootstrap starting.
I20260929 15:57:49.554040 28596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:49.555498 28596 log.cc:824] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:57:49 dist-test-slave-cq88 krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:49.558700 28596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b: No bootstrap required, opened a new log
I20260929 15:57:49.564785 28599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:49.543077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45482 (local address 127.23.222.126:36285)
0929 15:57:49.543968 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:49.543980 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:49.544024 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:57:49.545667 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:49.545690 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:49.546108 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:49.546467 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:49.548454 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:49.549780 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:49.551569 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:49.552325 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:49.552467 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:49.557390 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:49.557399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:49.557415 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:49.557450 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:49.560506 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:49.561249 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:49.561255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:49.561263 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:49.561462 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:49.561862 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:49.561866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:49.561868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:49.562418 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:49.562750 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:49.563225 (+ 475us) server_negotiation.cc:299] Negotiation successful
0929 15:57:49.563692 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":465,"threads_started":1}
I20260929 15:57:49.567328 28596 raft_consensus.cc:348] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc4d8bedcdd4a34a74392d5a659c37b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36285 } }
I20260929 15:57:49.567734 28596 raft_consensus.cc:374] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:49.567863 28596 raft_consensus.cc:729] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc4d8bedcdd4a34a74392d5a659c37b, State: Initialized, Role: FOLLOWER
I20260929 15:57:49.568481 28596 consensus_queue.cc:260] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc4d8bedcdd4a34a74392d5a659c37b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36285 } }
I20260929 15:57:49.568730 28596 raft_consensus.cc:388] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:49.568877 28596 raft_consensus.cc:482] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:49.569057 28596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:49.572324 28596 raft_consensus.cc:504] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc4d8bedcdd4a34a74392d5a659c37b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36285 } }
I20260929 15:57:49.572994 28596 leader_election.cc:302] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adc4d8bedcdd4a34a74392d5a659c37b; no voters:
I20260929 15:57:49.573896 28596 leader_election.cc:288] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:49.574147 28601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:49.575980 28601 raft_consensus.cc:686] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [term 1 LEADER]: Becoming Leader. State: Replica: adc4d8bedcdd4a34a74392d5a659c37b, State: Running, Role: LEADER
I20260929 15:57:49.576721 28601 consensus_queue.cc:237] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc4d8bedcdd4a34a74392d5a659c37b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36285 } }
I20260929 15:57:49.577389 28596 sys_catalog.cc:564] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:49.581096 28603 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc4d8bedcdd4a34a74392d5a659c37b. Latest consensus state: current_term: 1 leader_uuid: "adc4d8bedcdd4a34a74392d5a659c37b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc4d8bedcdd4a34a74392d5a659c37b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36285 } } }
I20260929 15:57:49.581329 28602 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adc4d8bedcdd4a34a74392d5a659c37b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc4d8bedcdd4a34a74392d5a659c37b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36285 } } }
I20260929 15:57:49.581666 28602 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:49.581664 28603 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:49.584307 28606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:49.593195 28606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:49.606280 28606 catalog_manager.cc:1347] Generated new cluster ID: 073b167b691e4f3bb0c1ccb6bf55c11d
I20260929 15:57:49.606415 28606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:49.624056 28606 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:49.626063 28606 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:49.636119 28606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b: Generated new TSK 0
I20260929 15:57:49.637288 28606 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:57:49.696403 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:36285
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:50.038339 28622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:50.038655 28622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:50.038754 28622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:50.046108 28622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:50.046233 28622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:50.046351 28622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:57:50.056299 28622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:50.057745 28622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:50.059113 28622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:50.072391 28627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:50.072338 28628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:50.072358 28630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:50.073732 28622 server_base.cc:1034] running on GCE node
I20260929 15:57:50.074494 28622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:50.075631 28622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:50.076870 28622 hybrid_clock.cc:648] HybridClock initialized: now 1790697470076818 us; error 41 us; skew 500 ppm
I20260929 15:57:50.077255 28622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:50 dist-test-slave-cq88 krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697470, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:50.080809 28622 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:57:50.082038 28622 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:50.082151 28622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:50.082471 28622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:50.085666 28622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "215ffd1315b24422a52d61acefd8bd61"
format_stamp: "Formatted at 2026-09-29 15:57:50 on dist-test-slave-cq88"
server_key: "5e27b11e9afcd0584e56cfc565759f3e"
server_key_iv: "3e95a9b82325d956264bbb4b6e3665a0"
server_key_version: "encryptionkey@0"
I20260929 15:57:50.086552 28622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "215ffd1315b24422a52d61acefd8bd61"
format_stamp: "Formatted at 2026-09-29 15:57:50 on dist-test-slave-cq88"
server_key: "5e27b11e9afcd0584e56cfc565759f3e"
server_key_iv: "3e95a9b82325d956264bbb4b6e3665a0"
server_key_version: "encryptionkey@0"
I20260929 15:57:50.091794 28622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:57:50.095468 28636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:50.096922 28622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:50.097129 28622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "215ffd1315b24422a52d61acefd8bd61"
format_stamp: "Formatted at 2026-09-29 15:57:50 on dist-test-slave-cq88"
server_key: "5e27b11e9afcd0584e56cfc565759f3e"
server_key_iv: "3e95a9b82325d956264bbb4b6e3665a0"
server_key_version: "encryptionkey@0"
I20260929 15:57:50.097369 28622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:50.133467 28622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:50.138146 28622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:50.138444 28622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:50.139609 28622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:50.141012 28622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:50.141100 28622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:50.141228 28622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:50.141328 28622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:50.175293 28622 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:35321
I20260929 15:57:50.175508 28748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:35321 every 8 connection(s)
I20260929 15:57:50.176262 28622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:50.179502 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 28622
I20260929 15:57:50.179739 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:50.180764 24441 external_mini_cluster.cc:1428] Setting key 740d9b34b0d6fa72647ce5ef4f5fb514
Sep 29 15:57:50 dist-test-slave-cq88 krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697470, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 15:57:50.204586 28751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:50.179451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:57903 (local address 127.23.222.126:36285)
0929 15:57:50.182712 (+ 3261us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:50.182721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:50.184416 (+ 1695us) server_negotiation.cc:407] Connection header received
0929 15:57:50.186364 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:50.186368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:50.186450 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:50.186540 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:50.189007 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:50.189804 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:50.192057 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:50.192787 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:50.192938 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:50.196863 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:50.196872 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:50.196880 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:50.196931 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:50.200857 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:50.201752 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:50.201762 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:50.201768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:50.201928 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:50.202419 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:50.202425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:50.202428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:50.202816 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:50.202979 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:50.203871 (+ 892us) server_negotiation.cc:299] Negotiation successful
0929 15:57:50.204124 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3104,"thread_start_us":2946,"threads_started":1}
I20260929 15:57:50.205361 28752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:50.180042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:36285 (local address 127.23.222.65:57903)
0929 15:57:50.183160 (+ 3118us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:50.183218 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:50.185994 (+ 2776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:50.186880 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:50.186896 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:50.187823 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:50.188754 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:50.188785 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:50.189997 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:50.190005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:50.191876 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:50.191890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:50.193093 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:50.193099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:50.193325 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:50.194146 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:50.194201 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:50.196568 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:50.201066 (+ 4498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:50.201075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:50.201091 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:50.201591 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:50.202050 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:50.202056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:50.202059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:50.202292 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:50.202956 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:50.202962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:50.203407 (+ 445us) client_negotiation.cc:769] Sending connection context
0929 15:57:50.203676 (+ 269us) client_negotiation.cc:240] Negotiation successful
0929 15:57:50.204110 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2759,"thread_start_us":2526,"threads_started":1}
I20260929 15:57:50.207194 28749 heartbeater.cc:347] Connected to a master server at 127.23.222.126:36285
I20260929 15:57:50.207574 28749 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
I20260929 15:57:50.208488 28749 heartbeater.cc:511] Master 127.23.222.126:36285 requested a full tablet report, sending...
I20260929 15:57:50.211067 28561 ts_manager.cc:194] Registered new tserver with Master: 215ffd1315b24422a52d61acefd8bd61 (127.23.222.65:35321)
I20260929 15:57:50.213250 28561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:57903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:57:50.235605 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:36285
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:50.576763 28755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:50.577085 28755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:50.577188 28755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:50.584872 28755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:50.584993 28755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:50.585129 28755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:57:50.595520 28755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:50.596825 28755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:50.598049 28755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:50.611308 28760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:50.611349 28761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:50.611374 28763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:50.611958 28755 server_base.cc:1034] running on GCE node
I20260929 15:57:50.612635 28755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:50.613675 28755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:50.614900 28755 hybrid_clock.cc:648] HybridClock initialized: now 1790697470614846 us; error 50 us; skew 500 ppm
I20260929 15:57:50.615284 28755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:50 dist-test-slave-cq88 krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697470, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:50.618314 28755 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:57:50.619535 28755 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:50.619666 28755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:50.619974 28755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:50.622906 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "a15fe1d3b5164adca9a56ad3d8d0a336"
format_stamp: "Formatted at 2026-09-29 15:57:50 on dist-test-slave-cq88"
server_key: "cb1c8637b018ab24746cb0343e7a9247"
server_key_iv: "1225beed38a08352baaf386955458969"
server_key_version: "encryptionkey@0"
I20260929 15:57:50.623818 28755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "a15fe1d3b5164adca9a56ad3d8d0a336"
format_stamp: "Formatted at 2026-09-29 15:57:50 on dist-test-slave-cq88"
server_key: "cb1c8637b018ab24746cb0343e7a9247"
server_key_iv: "1225beed38a08352baaf386955458969"
server_key_version: "encryptionkey@0"
I20260929 15:57:50.628891 28755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:57:50.632403 28769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:50.633908 28755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:50.634102 28755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "a15fe1d3b5164adca9a56ad3d8d0a336"
format_stamp: "Formatted at 2026-09-29 15:57:50 on dist-test-slave-cq88"
server_key: "cb1c8637b018ab24746cb0343e7a9247"
server_key_iv: "1225beed38a08352baaf386955458969"
server_key_version: "encryptionkey@0"
I20260929 15:57:50.634342 28755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:50.663437 28755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:50.667676 28755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:50.667968 28755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:50.668987 28755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:50.670336 28755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:50.670459 28755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:50.670598 28755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:50.670688 28755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:50.704738 28755 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:36205
I20260929 15:57:50.704826 28881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:36205 every 8 connection(s)
I20260929 15:57:50.705689 28755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:50.715642 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 28755
I20260929 15:57:50.715868 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:50.716291 24441 external_mini_cluster.cc:1428] Setting key e136ac1d9a32810e5e469a1e1450b86d
Sep 29 15:57:50 dist-test-slave-cq88 krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697470, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:50.727059 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:50.708408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:57769 (local address 127.23.222.126:36285)
0929 15:57:50.708898 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:50.708903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:50.709905 (+ 1002us) server_negotiation.cc:407] Connection header received
0929 15:57:50.712000 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:50.712005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:50.712085 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:50.712189 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:50.713893 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:50.714641 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:50.716587 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:50.717172 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:50.717320 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:50.720875 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:50.720880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:50.720884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:50.720917 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:50.723537 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:50.724614 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:50.724619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:50.724621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:50.724788 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:50.725280 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:50.725283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:50.725285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:50.725563 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:50.725686 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:50.726594 (+ 908us) server_negotiation.cc:299] Negotiation successful
0929 15:57:50.726791 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":262,"threads_started":1}
I20260929 15:57:50.728673 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:50.709006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:36285 (local address 127.23.222.66:57769)
0929 15:57:50.709727 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:50.709769 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:50.711686 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:50.712483 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:50.712492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:50.713062 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:50.713663 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:50.713683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:50.714838 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:50.714842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:50.716409 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:50.716427 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:50.717418 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:50.717428 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:50.717671 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:50.718382 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:50.718411 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:50.720629 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:50.723735 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:50.723745 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:50.723766 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:50.724444 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:50.724916 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:50.724924 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:50.724931 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:50.725165 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:50.725696 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:50.725707 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:50.726202 (+ 495us) client_negotiation.cc:769] Sending connection context
0929 15:57:50.726532 (+ 330us) client_negotiation.cc:240] Negotiation successful
0929 15:57:50.727126 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":289,"threads_started":1}
I20260929 15:57:50.730321 28882 heartbeater.cc:347] Connected to a master server at 127.23.222.126:36285
I20260929 15:57:50.730655 28882 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:50.731406 28882 heartbeater.cc:511] Master 127.23.222.126:36285 requested a full tablet report, sending...
I20260929 15:57:50.732932 28561 ts_manager.cc:194] Registered new tserver with Master: a15fe1d3b5164adca9a56ad3d8d0a336 (127.23.222.66:36205)
I20260929 15:57:50.733744 28561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:57769
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:57:50.762881 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:36285
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:36149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:57:51.095809 28888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:51.096177 28888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:51.096279 28888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:51.104295 28888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:51.104416 28888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:51.104534 28888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:57:51.114355 28888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:36149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:51.115959 28888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:51.117264 28888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:51.130439 28893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:51.130402 28896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:51.130385 28894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:51.131072 28888 server_base.cc:1034] running on GCE node
I20260929 15:57:51.131870 28888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:51.132848 28888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:51.134084 28888 hybrid_clock.cc:648] HybridClock initialized: now 1790697471134038 us; error 42 us; skew 500 ppm
I20260929 15:57:51.134440 28888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:57:51 dist-test-slave-cq88 krb5kdc[28517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697471, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:57:51.137701 28888 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:57:51.138942 28888 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:51.139070 28888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:51.139462 28888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:51.142486 28888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "254db3b4578b46c9a19b9dd452085254"
format_stamp: "Formatted at 2026-09-29 15:57:51 on dist-test-slave-cq88"
server_key: "80805890be2a52e8ff0809058b31ec1e"
server_key_iv: "3f94966e8e1c387204ea59a798703262"
server_key_version: "encryptionkey@0"
I20260929 15:57:51.143337 28888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "254db3b4578b46c9a19b9dd452085254"
format_stamp: "Formatted at 2026-09-29 15:57:51 on dist-test-slave-cq88"
server_key: "80805890be2a52e8ff0809058b31ec1e"
server_key_iv: "3f94966e8e1c387204ea59a798703262"
server_key_version: "encryptionkey@0"
I20260929 15:57:51.148375 28888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 15:57:51.151865 28902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:51.153348 28888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:51.153527 28888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "254db3b4578b46c9a19b9dd452085254"
format_stamp: "Formatted at 2026-09-29 15:57:51 on dist-test-slave-cq88"
server_key: "80805890be2a52e8ff0809058b31ec1e"
server_key_iv: "3f94966e8e1c387204ea59a798703262"
server_key_version: "encryptionkey@0"
I20260929 15:57:51.153762 28888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:51.193037 28888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:51.197717 28888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:51.198011 28888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:51.199018 28888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:51.200433 28888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:51.200639 28888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:51.200788 28888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:51.200875 28888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:51.216686 28749 heartbeater.cc:503] Master 127.23.222.126:36285 was elected leader, sending a full tablet report...
I20260929 15:57:51.235684 28888 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:41247
I20260929 15:57:51.235787 29014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:41247 every 8 connection(s)
I20260929 15:57:51.236603 28888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:51.243693 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 28888
I20260929 15:57:51.243909 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:51.244345 24441 external_mini_cluster.cc:1428] Setting key aaaa72ba940078c2d522232fa11bc634
Sep 29 15:57:51 dist-test-slave-cq88 krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697471, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:57:51.257876 29017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.239286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:43089 (local address 127.23.222.126:36285)
0929 15:57:51.239745 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.239750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.240776 (+ 1026us) server_negotiation.cc:407] Connection header received
0929 15:57:51.242624 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.242629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.242738 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.242855 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:51.244736 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.245701 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.247381 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.247870 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.247970 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.251911 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:51.251917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:51.251920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:51.251956 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:51.254746 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.255577 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.255581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.255583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.255706 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.256081 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.256084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.256086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.256366 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:51.256473 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.257260 (+ 787us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.257498 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":220,"threads_started":1}
I20260929 15:57:51.259006 29018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.239845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:36285 (local address 127.23.222.67:43089)
0929 15:57:51.240602 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:51.240640 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:51.242267 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:51.243081 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:51.243089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:51.243756 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:51.244513 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.244535 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.245881 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.245891 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.247187 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.247200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.248125 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.248133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.248421 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.249132 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:57:51.249156 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:51.251610 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:51.254906 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:51.254913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:51.254928 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:51.255459 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:51.255800 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:57:51.255803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:57:51.255805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:57:51.255983 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:57:51.256467 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:51.256472 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:51.256827 (+ 355us) client_negotiation.cc:769] Sending connection context
0929 15:57:51.257108 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 15:57:51.257722 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":298,"threads_started":1}
I20260929 15:57:51.260658 29015 heartbeater.cc:347] Connected to a master server at 127.23.222.126:36285
I20260929 15:57:51.261008 29015 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:51.261739 29015 heartbeater.cc:511] Master 127.23.222.126:36285 requested a full tablet report, sending...
I20260929 15:57:51.263218 28561 ts_manager.cc:194] Registered new tserver with Master: 254db3b4578b46c9a19b9dd452085254 (127.23.222.67:41247)
I20260929 15:57:51.264196 28561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:43089
I20260929 15:57:51.272979 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:57:51.290571 29017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.277725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45494 (local address 127.23.222.126:36285)
0929 15:57:51.277893 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.277898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.278459 (+ 561us) server_negotiation.cc:407] Connection header received
0929 15:57:51.278635 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.278643 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.278763 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.278850 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:51.280289 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.281075 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.282277 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.282654 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.282753 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.285231 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:51.285236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:51.285240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:51.285276 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:51.288180 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.288824 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.288828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.288830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.288945 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.289248 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.289251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.289253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.289568 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:51.289776 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.290100 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.290294 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:57:51.294790 28561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:51.301854 28561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:51.326272 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.314808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.23.222.66:36205)
0929 15:57:51.315437 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.315446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.316150 (+ 704us) server_negotiation.cc:407] Connection header received
0929 15:57:51.316532 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.316543 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.316868 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.317090 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:51.318130 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.320139 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.323462 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.324946 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.325091 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.325322 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.325689 (+ 367us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.325900 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":309,"threads_started":1}
I20260929 15:57:51.327303 29028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.314651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:36205 (local address 127.0.0.1:38410)
0929 15:57:51.315994 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:51.316031 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:51.316294 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:51.317506 (+ 1212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:51.317517 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:51.317559 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:51.317924 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.317940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.320385 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.320393 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.323277 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.323293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.325146 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.325155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.325387 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.325445 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 15:57:51.326728 (+ 1283us) client_negotiation.cc:240] Negotiation successful
0929 15:57:51.326934 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1108,"thread_start_us":963,"threads_started":1}
I20260929 15:57:51.331480 28817 tablet_service.cc:1467] Processing CreateTablet for tablet 5c597c1636b74840a978d9bd165d1c87 (DEFAULT_TABLE table=test-table [id=fa21903ab3cd43899f80d004440798b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:51.332527 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.316746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.23.222.65:35321)
0929 15:57:51.323808 (+ 7062us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.323822 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.324584 (+ 762us) server_negotiation.cc:407] Connection header received
0929 15:57:51.324710 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.324721 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.325035 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.325240 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:51.326133 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.327683 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.329979 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.331445 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.331626 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.331857 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.332048 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.332208 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6849,"thread_start_us":6692,"threads_started":1}
I20260929 15:57:51.332870 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.316458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:35321 (local address 127.0.0.1:43166)
0929 15:57:51.324258 (+ 7800us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:51.324282 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:51.324435 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:51.325469 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:51.325473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:51.325497 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:51.325935 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.325948 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.327827 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.327834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.329801 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.329820 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.331660 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.331666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.331841 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.331876 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:57:51.332008 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 15:57:51.332154 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7655,"mutex_wait_us":25,"thread_start_us":7998,"threads_started":1}
I20260929 15:57:51.332909 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.315193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:41247 (local address 127.0.0.1:41764)
0929 15:57:51.323394 (+ 8201us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:51.323415 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:51.323518 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:51.326110 (+ 2592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:51.326114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:51.326138 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:51.326548 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.326559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.328631 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.328638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.330186 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.330200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.331991 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.331998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.332173 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.332212 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:57:51.332292 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 15:57:51.332435 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8069,"mutex_wait_us":22,"thread_start_us":7889,"threads_started":1}
I20260929 15:57:51.333127 28817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c597c1636b74840a978d9bd165d1c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:51.333678 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.315641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41764 (local address 127.23.222.67:41247)
0929 15:57:51.325482 (+ 9841us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.325503 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.325540 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:57:51.325647 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.325654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.325884 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.326772 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:51.326999 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.328489 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.330352 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.331751 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.332855 (+ 1104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.333092 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.333167 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.333359 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9333,"thread_start_us":9169,"threads_started":1}
I20260929 15:57:51.336962 28950 tablet_service.cc:1467] Processing CreateTablet for tablet 5c597c1636b74840a978d9bd165d1c87 (DEFAULT_TABLE table=test-table [id=fa21903ab3cd43899f80d004440798b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:51.336968 28684 tablet_service.cc:1467] Processing CreateTablet for tablet 5c597c1636b74840a978d9bd165d1c87 (DEFAULT_TABLE table=test-table [id=fa21903ab3cd43899f80d004440798b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:51.338411 28684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c597c1636b74840a978d9bd165d1c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:51.338428 28950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c597c1636b74840a978d9bd165d1c87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:51.345785 29034 tablet_bootstrap.cc:492] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: Bootstrap starting.
I20260929 15:57:51.349051 29034 tablet_bootstrap.cc:654] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:51.350152 29034 log.cc:824] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:51.354009 29034 tablet_bootstrap.cc:492] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: No bootstrap required, opened a new log
I20260929 15:57:51.354336 29034 ts_tablet_manager.cc:1397] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 15:57:51.355041 29037 tablet_bootstrap.cc:492] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: Bootstrap starting.
I20260929 15:57:51.359115 29037 tablet_bootstrap.cc:654] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:51.360513 29037 log.cc:824] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:51.361804 29036 tablet_bootstrap.cc:492] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: Bootstrap starting.
I20260929 15:57:51.365074 29034 raft_consensus.cc:348] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } }
I20260929 15:57:51.365595 29034 raft_consensus.cc:374] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:51.365705 29034 raft_consensus.cc:729] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a15fe1d3b5164adca9a56ad3d8d0a336, State: Initialized, Role: FOLLOWER
I20260929 15:57:51.366360 29037 tablet_bootstrap.cc:492] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: No bootstrap required, opened a new log
I20260929 15:57:51.366465 29036 tablet_bootstrap.cc:654] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:51.366389 29034 consensus_queue.cc:260] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } }
I20260929 15:57:51.366706 29037 ts_tablet_manager.cc:1397] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 15:57:51.367707 28882 heartbeater.cc:503] Master 127.23.222.126:36285 was elected leader, sending a full tablet report...
I20260929 15:57:51.367930 29036 log.cc:824] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:51.368395 29034 ts_tablet_manager.cc:1428] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260929 15:57:51.371101 29036 tablet_bootstrap.cc:492] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: No bootstrap required, opened a new log
I20260929 15:57:51.371433 29036 ts_tablet_manager.cc:1397] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 15:57:51.377555 29037 raft_consensus.cc:348] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } }
I20260929 15:57:51.378121 29037 raft_consensus.cc:374] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:51.378244 29037 raft_consensus.cc:729] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254db3b4578b46c9a19b9dd452085254, State: Initialized, Role: FOLLOWER
I20260929 15:57:51.378933 29037 consensus_queue.cc:260] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } }
I20260929 15:57:51.379568 29036 raft_consensus.cc:348] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } }
I20260929 15:57:51.380060 29036 raft_consensus.cc:374] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:51.380191 29036 raft_consensus.cc:729] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 215ffd1315b24422a52d61acefd8bd61, State: Initialized, Role: FOLLOWER
I20260929 15:57:51.380200 29015 heartbeater.cc:503] Master 127.23.222.126:36285 was elected leader, sending a full tablet report...
I20260929 15:57:51.380862 29037 ts_tablet_manager.cc:1428] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 15:57:51.380805 29036 consensus_queue.cc:260] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } }
I20260929 15:57:51.382562 29036 ts_tablet_manager.cc:1428] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20260929 15:57:51.428841 28750 tablet.cc:2369] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:57:51.457772 28883 tablet.cc:2369] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:51.470335 29042 raft_consensus.cc:482] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:51.470659 29042 raft_consensus.cc:504] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } }
I20260929 15:57:51.472378 29042 leader_election.cc:288] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a15fe1d3b5164adca9a56ad3d8d0a336 (127.23.222.66:36205), 254db3b4578b46c9a19b9dd452085254 (127.23.222.67:41247)
I20260929 15:57:51.479724 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.472816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:51263 (local address 127.23.222.67:41247)
0929 15:57:51.473022 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.473029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.473841 (+ 812us) server_negotiation.cc:407] Connection header received
0929 15:57:51.474092 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.474106 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.474183 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.474291 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:51.475306 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.476112 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.477885 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.478720 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.478899 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.478985 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.479313 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.479441 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:57:51.479744 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.472928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:46181 (local address 127.23.222.66:36205)
0929 15:57:51.473124 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.473132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.473840 (+ 708us) server_negotiation.cc:407] Connection header received
0929 15:57:51.474097 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.474102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.474182 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.474288 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:57:51.475228 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.476025 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.477906 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.478764 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.478942 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.479021 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.479355 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.479481 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:57:51.479916 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.472752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:41247 (local address 127.23.222.65:51263)
0929 15:57:51.473703 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:51.473719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:51.473883 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:51.474352 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:51.474358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:51.474403 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:51.475060 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.475069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.476273 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.476278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.477730 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.477745 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.478953 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.478960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.479127 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.479152 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 15:57:51.479285 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 15:57:51.479427 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":30,"spinlock_wait_cycles":59392,"thread_start_us":462,"threads_started":1}
I20260929 15:57:51.479943 29044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.472661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:36205 (local address 127.23.222.65:46181)
0929 15:57:51.473639 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:51.473660 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:51.473856 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:51.474352 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:51.474358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:51.474391 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:57:51.475038 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.475058 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.476159 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.476166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.477729 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:51.477752 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.479001 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.479007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:51.479173 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.479196 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 15:57:51.479327 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 15:57:51.479465 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":10,"thread_start_us":716,"threads_started":1}
I20260929 15:57:51.480969 28837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c597c1636b74840a978d9bd165d1c87" candidate_uuid: "215ffd1315b24422a52d61acefd8bd61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" is_pre_election: true
I20260929 15:57:51.481045 28970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c597c1636b74840a978d9bd165d1c87" candidate_uuid: "215ffd1315b24422a52d61acefd8bd61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254db3b4578b46c9a19b9dd452085254" is_pre_election: true
I20260929 15:57:51.481514 28837 raft_consensus.cc:2454] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 215ffd1315b24422a52d61acefd8bd61 in term 0.
I20260929 15:57:51.481551 28970 raft_consensus.cc:2454] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 215ffd1315b24422a52d61acefd8bd61 in term 0.
I20260929 15:57:51.482193 28637 leader_election.cc:302] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 215ffd1315b24422a52d61acefd8bd61, a15fe1d3b5164adca9a56ad3d8d0a336; no voters:
I20260929 15:57:51.482648 29042 raft_consensus.cc:2781] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:51.482796 29042 raft_consensus.cc:482] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:51.482930 29042 raft_consensus.cc:3037] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:51.485348 29042 raft_consensus.cc:504] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } }
I20260929 15:57:51.485971 29042 leader_election.cc:288] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [CANDIDATE]: Term 1 election: Requested vote from peers a15fe1d3b5164adca9a56ad3d8d0a336 (127.23.222.66:36205), 254db3b4578b46c9a19b9dd452085254 (127.23.222.67:41247)
I20260929 15:57:51.486541 28837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c597c1636b74840a978d9bd165d1c87" candidate_uuid: "215ffd1315b24422a52d61acefd8bd61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336"
I20260929 15:57:51.486534 28970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c597c1636b74840a978d9bd165d1c87" candidate_uuid: "215ffd1315b24422a52d61acefd8bd61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "254db3b4578b46c9a19b9dd452085254"
I20260929 15:57:51.486759 28970 raft_consensus.cc:3037] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:51.486760 28837 raft_consensus.cc:3037] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [term 0 FOLLOWER]: Advancing to term 1
W20260929 15:57:51.488149 29016 tablet.cc:2369] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:51.489248 28837 raft_consensus.cc:2454] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 215ffd1315b24422a52d61acefd8bd61 in term 1.
I20260929 15:57:51.489632 28970 raft_consensus.cc:2454] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 215ffd1315b24422a52d61acefd8bd61 in term 1.
I20260929 15:57:51.489717 28637 leader_election.cc:302] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 215ffd1315b24422a52d61acefd8bd61, a15fe1d3b5164adca9a56ad3d8d0a336; no voters:
I20260929 15:57:51.490090 29042 raft_consensus.cc:2781] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:51.490545 29042 raft_consensus.cc:686] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 1 LEADER]: Becoming Leader. State: Replica: 215ffd1315b24422a52d61acefd8bd61, State: Running, Role: LEADER
I20260929 15:57:51.491016 29042 consensus_queue.cc:237] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } }
I20260929 15:57:51.495963 28561 catalog_manager.cc:5495] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 215ffd1315b24422a52d61acefd8bd61 (127.23.222.65). New cstate: current_term: 1 leader_uuid: "215ffd1315b24422a52d61acefd8bd61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "215ffd1315b24422a52d61acefd8bd61" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35321 } health_report { overall_health: HEALTHY } } }
I20260929 15:57:51.517659 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.512293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.23.222.65:35321)
0929 15:57:51.512486 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.512492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.512511 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:57:51.512738 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.512743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.512829 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.512983 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:57:51.513348 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.514061 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.515481 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.515914 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.516043 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.516543 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:57:51.517062 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:57:51.517233 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.517310 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.517393 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 15:57:51.526062 28837 raft_consensus.cc:1261] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [term 1 FOLLOWER]: Refusing update from remote peer 215ffd1315b24422a52d61acefd8bd61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:51.526091 28970 raft_consensus.cc:1261] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [term 1 FOLLOWER]: Refusing update from remote peer 215ffd1315b24422a52d61acefd8bd61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:51.527122 29045 consensus_queue.cc:1035] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "254db3b4578b46c9a19b9dd452085254" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 41247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:51.527513 29042 consensus_queue.cc:1035] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a15fe1d3b5164adca9a56ad3d8d0a336" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:51.565011 29050 mvcc.cc:205] Tried to move back new op lower bound from 7334696843360575488 to 7334696843230867456. Current Snapshot: MvccSnapshot[applied={T|T < 7334696843360575488}]
I20260929 15:57:51.571377 28561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:51.571697 28561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:51.576020 28561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adc4d8bedcdd4a34a74392d5a659c37b: Sending DeleteTablet for 3 replicas of tablet 5c597c1636b74840a978d9bd165d1c87
I20260929 15:57:51.577369 28950 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c597c1636b74840a978d9bd165d1c87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:51 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:41764
I20260929 15:57:51.577410 28817 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c597c1636b74840a978d9bd165d1c87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:51 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:38410
I20260929 15:57:51.577445 28684 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c597c1636b74840a978d9bd165d1c87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:51 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:43166
I20260929 15:57:51.579939 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 28622
I20260929 15:57:51.606638 29059 tablet_replica.cc:330] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: stopping tablet replica
I20260929 15:57:51.607964 29060 tablet_replica.cc:330] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: stopping tablet replica
I20260929 15:57:51.608476 29060 raft_consensus.cc:2229] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:51.609056 29060 raft_consensus.cc:2258] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:51.608477 29059 raft_consensus.cc:2229] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:51.609694 29059 raft_consensus.cc:2258] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:51.609993 29058 tablet_replica.cc:330] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: stopping tablet replica
I20260929 15:57:51.610451 29058 raft_consensus.cc:2229] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:51.611191 29058 raft_consensus.cc:2258] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:51.611716 29060 ts_tablet_manager.cc:1905] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:51.613438 29059 ts_tablet_manager.cc:1905] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:51.613895 29058 ts_tablet_manager.cc:1905] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:51.619289 29060 ts_tablet_manager.cc:1918] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:51.619499 29060 log.cc:1196] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/5c597c1636b74840a978d9bd165d1c87
Sep 29 15:57:51 dist-test-slave-cq88 krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:57:51.620054 29060 ts_tablet_manager.cc:1939] T 5c597c1636b74840a978d9bd165d1c87 P 254db3b4578b46c9a19b9dd452085254: Deleting consensus metadata
I20260929 15:57:51.621352 29059 ts_tablet_manager.cc:1918] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:51.621599 29059 log.cc:1196] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/5c597c1636b74840a978d9bd165d1c87
I20260929 15:57:51.622108 28548 catalog_manager.cc:4841] TS 254db3b4578b46c9a19b9dd452085254 (127.23.222.67:41247): tablet 5c597c1636b74840a978d9bd165d1c87 (table test-table [id=fa21903ab3cd43899f80d004440798b2]) successfully deleted
I20260929 15:57:51.622324 29059 ts_tablet_manager.cc:1939] T 5c597c1636b74840a978d9bd165d1c87 P a15fe1d3b5164adca9a56ad3d8d0a336: Deleting consensus metadata
I20260929 15:57:51.624325 28545 catalog_manager.cc:4841] TS a15fe1d3b5164adca9a56ad3d8d0a336 (127.23.222.66:36205): tablet 5c597c1636b74840a978d9bd165d1c87 (table test-table [id=fa21903ab3cd43899f80d004440798b2]) successfully deleted
I20260929 15:57:51.628036 29058 ts_tablet_manager.cc:1918] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:51.628218 29058 log.cc:1196] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/5c597c1636b74840a978d9bd165d1c87
I20260929 15:57:51.628602 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.582124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.23.222.65:35321)
0929 15:57:51.582378 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.582387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.605436 (+ 23049us) server_negotiation.cc:407] Connection header received
0929 15:57:51.607040 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.607048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.607152 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.607266 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:51.611022 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.612644 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.614476 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.615186 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.615324 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.620937 (+ 5613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:51.620958 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:51.620984 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:51.621044 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:51.625227 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.626029 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.626037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.626040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.626202 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.626611 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.626615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.626617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.626987 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:51.627705 (+ 718us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.627869 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.628129 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 15:57:51.628768 29058 ts_tablet_manager.cc:1939] T 5c597c1636b74840a978d9bd165d1c87 P 215ffd1315b24422a52d61acefd8bd61: Deleting consensus metadata
I20260929 15:57:51.629288 28744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:51.738716 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 28622
I20260929 15:57:51.738857 28545 catalog_manager.cc:4841] TS 215ffd1315b24422a52d61acefd8bd61 (127.23.222.65:35321): tablet 5c597c1636b74840a978d9bd165d1c87 (table test-table [id=fa21903ab3cd43899f80d004440798b2]) successfully deleted
I20260929 15:57:51.756623 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 28755
Sep 29 15:57:51 dist-test-slave-cq88 krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:57:51.771941 29029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.757374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.23.222.66:36205)
0929 15:57:51.757564 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.757570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.757593 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:57:51.757866 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.757871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.757937 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.758028 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:51.759406 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.760330 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.761680 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.762099 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.762208 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.766542 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:51.766556 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:51.766573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:51.766609 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:51.769367 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.770035 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.770042 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.770045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.770156 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.770480 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.770483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.770485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.770833 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:51.771044 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.771441 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.771604 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 15:57:51.772612 28877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:51.865054 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 28755
I20260929 15:57:51.882746 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 28888
Sep 29 15:57:51 dist-test-slave-cq88 krb5kdc[28517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:57:51.899093 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:51.883617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.23.222.67:41247)
0929 15:57:51.883816 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:51.883821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:51.883916 (+ 95us) server_negotiation.cc:407] Connection header received
0929 15:57:51.884046 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:51.884051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:51.884120 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:51.884229 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:51.885664 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.886540 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.887917 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:51.888350 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:51.888459 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:51.893094 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:51.893111 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:51.893127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:57:51.893168 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:51.896143 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.896848 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.896855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.896857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.896989 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:57:51.897570 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:57:51.897574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:57:51.897576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:57:51.897978 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:51.898227 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:51.898521 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 15:57:51.898708 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:57:51.900039 29010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:51.988933 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 28888
I20260929 15:57:52.005807 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 28530
I20260929 15:57:52.007098 28591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:52.116425 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 28530
2026-09-29T15:57:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4358 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T15:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:52Z Disabled control of system clock
I20260929 15:57:52.188798 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37537
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:37537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 15:57:52.520650 29069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:52.520951 29069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:52.521102 29069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:52.528489 29069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:52.528602 29069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:52.528695 29069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:52.528800 29069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:52.528884 29069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:57:52.538424 29069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:37537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:52.539780 29069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:52.541591 29069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:52.553622 29074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:52.553711 29075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:52.553732 29077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:52.555017 29069 server_base.cc:1034] running on GCE node
I20260929 15:57:52.555894 29069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:52.557494 29069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:52.558686 29069 hybrid_clock.cc:648] HybridClock initialized: now 1790697472558647 us; error 68 us; skew 500 ppm
I20260929 15:57:52.559099 29069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:57:52.566825 29069 webserver.cc:460] Webserver started at http://127.23.222.126:35263/ using document root <none> and password file <none>
I20260929 15:57:52.567754 29069 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:52.567855 29069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:52.568315 29069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:52.571920 29069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "dcd787bd69cb428fa696e8ecc5b511c1"
format_stamp: "Formatted at 2026-09-29 15:57:52 on dist-test-slave-cq88"
server_key: "554d860a09f8dd7c75b8243e47191024"
server_key_iv: "1030e266cc7a0afc44851a9e3b9b7b95"
server_key_version: "encryptionkey@0"
I20260929 15:57:52.572853 29069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "dcd787bd69cb428fa696e8ecc5b511c1"
format_stamp: "Formatted at 2026-09-29 15:57:52 on dist-test-slave-cq88"
server_key: "554d860a09f8dd7c75b8243e47191024"
server_key_iv: "1030e266cc7a0afc44851a9e3b9b7b95"
server_key_version: "encryptionkey@0"
I20260929 15:57:52.578663 29069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:57:52.582444 29083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:52.584177 29069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:57:52.584363 29069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "dcd787bd69cb428fa696e8ecc5b511c1"
format_stamp: "Formatted at 2026-09-29 15:57:52 on dist-test-slave-cq88"
server_key: "554d860a09f8dd7c75b8243e47191024"
server_key_iv: "1030e266cc7a0afc44851a9e3b9b7b95"
server_key_version: "encryptionkey@0"
I20260929 15:57:52.584619 29069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:52.591316 29069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:52.592181 29069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:52.592501 29069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:52.610956 29069 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:37537
I20260929 15:57:52.610980 29134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:37537 every 8 connection(s)
I20260929 15:57:52.612396 29069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:52.616343 29135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:52.618958 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 29069
I20260929 15:57:52.619167 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:52.620009 24441 external_mini_cluster.cc:1428] Setting key 7f67ac2023d2f7565f920e146d333a0e
I20260929 15:57:52.629026 29138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:52.622345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.23.222.126:37537)
0929 15:57:52.623527 (+ 1182us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:52.623539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:52.623582 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:57:52.625200 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:52.625223 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:52.625670 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:52.626114 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:52.627291 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:52.627299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:52.627319 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:52.627335 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:52.627398 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:52.627426 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:52.627664 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:52.627820 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 15:57:52.628083 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":401,"threads_started":1}
I20260929 15:57:52.629302 29135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1: Bootstrap starting.
I20260929 15:57:52.632849 29135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:52.633901 29135 log.cc:824] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:52.636966 29135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1: No bootstrap required, opened a new log
I20260929 15:57:52.644761 29135 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd787bd69cb428fa696e8ecc5b511c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37537 } }
I20260929 15:57:52.645200 29135 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:52.645311 29135 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd787bd69cb428fa696e8ecc5b511c1, State: Initialized, Role: FOLLOWER
I20260929 15:57:52.645959 29135 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd787bd69cb428fa696e8ecc5b511c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37537 } }
I20260929 15:57:52.646210 29135 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:52.646353 29135 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:52.646516 29135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:52.649034 29135 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd787bd69cb428fa696e8ecc5b511c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37537 } }
I20260929 15:57:52.649570 29135 leader_election.cc:302] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcd787bd69cb428fa696e8ecc5b511c1; no voters:
I20260929 15:57:52.650197 29135 leader_election.cc:288] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:52.650368 29140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:52.651216 29140 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [term 1 LEADER]: Becoming Leader. State: Replica: dcd787bd69cb428fa696e8ecc5b511c1, State: Running, Role: LEADER
I20260929 15:57:52.651798 29140 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd787bd69cb428fa696e8ecc5b511c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37537 } }
I20260929 15:57:52.652491 29135 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:52.654795 29142 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcd787bd69cb428fa696e8ecc5b511c1. Latest consensus state: current_term: 1 leader_uuid: "dcd787bd69cb428fa696e8ecc5b511c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd787bd69cb428fa696e8ecc5b511c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37537 } } }
I20260929 15:57:52.655373 29142 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:52.654897 29141 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcd787bd69cb428fa696e8ecc5b511c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd787bd69cb428fa696e8ecc5b511c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37537 } } }
I20260929 15:57:52.655609 29141 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:52.656699 29149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:52.662874 29149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:52.672255 29149 catalog_manager.cc:1347] Generated new cluster ID: 3446e0a5622c405db3d48138f1548ad0
I20260929 15:57:52.672392 29149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:52.706694 29149 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:52.707301 29149 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:52.716097 29149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1: Generated new TSK 0
I20260929 15:57:52.717098 29149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:52.728636 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37537
--builtin_ntp_servers=127.23.222.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 15:57:53.087137 29159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:53.087456 29159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:53.087607 29159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:53.095052 29159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:53.095170 29159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:53.095366 29159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:57:53.104748 29159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:53.106134 29159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:53.107952 29159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:53.121496 29167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:53.121536 29165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:53.121786 29164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:53.122758 29159 server_base.cc:1034] running on GCE node
I20260929 15:57:53.123464 29159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:53.124454 29159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:53.125696 29159 hybrid_clock.cc:648] HybridClock initialized: now 1790697473125609 us; error 69 us; skew 500 ppm
I20260929 15:57:53.126062 29159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:57:53.128621 29159 webserver.cc:460] Webserver started at http://127.23.222.65:46187/ using document root <none> and password file <none>
I20260929 15:57:53.129354 29159 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:53.129469 29159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:53.129835 29159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:53.132879 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "769a3ffd5e64433e990b5608ba618aee"
format_stamp: "Formatted at 2026-09-29 15:57:53 on dist-test-slave-cq88"
server_key: "f3e03432b7f597a0b3d2dc8536bbc35d"
server_key_iv: "15f775ade55d42b48e493eafa8c570c4"
server_key_version: "encryptionkey@0"
I20260929 15:57:53.133729 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "769a3ffd5e64433e990b5608ba618aee"
format_stamp: "Formatted at 2026-09-29 15:57:53 on dist-test-slave-cq88"
server_key: "f3e03432b7f597a0b3d2dc8536bbc35d"
server_key_iv: "15f775ade55d42b48e493eafa8c570c4"
server_key_version: "encryptionkey@0"
I20260929 15:57:53.139129 29159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:57:53.142755 29173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:53.144201 29159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 15:57:53.144387 29159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "769a3ffd5e64433e990b5608ba618aee"
format_stamp: "Formatted at 2026-09-29 15:57:53 on dist-test-slave-cq88"
server_key: "f3e03432b7f597a0b3d2dc8536bbc35d"
server_key_iv: "15f775ade55d42b48e493eafa8c570c4"
server_key_version: "encryptionkey@0"
I20260929 15:57:53.144621 29159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:53.150704 29159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:53.151602 29159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:53.151875 29159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:53.152936 29159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:53.154541 29159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:53.154664 29159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:53.154794 29159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:53.154879 29159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:53.188812 29159 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:36743
I20260929 15:57:53.188853 29285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:36743 every 8 connection(s)
I20260929 15:57:53.190274 29159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:57:53.192479 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 29159
I20260929 15:57:53.192673 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:57:53.193464 24441 external_mini_cluster.cc:1428] Setting key d9ca1e189ddfbd8a99f8f6af1c91e977
I20260929 15:57:53.196171 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37537
--builtin_ntp_servers=127.23.222.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 15:57:53.199222 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:53.193131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:43993 (local address 127.23.222.126:37537)
0929 15:57:53.193656 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:53.193662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:53.195059 (+ 1397us) server_negotiation.cc:407] Connection header received
0929 15:57:53.196900 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:53.196905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:53.196974 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:53.197100 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:53.198435 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:53.198441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:53.198446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:53.198465 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:53.198517 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:53.198529 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:53.198660 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:53.198857 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 15:57:53.198927 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":240,"threads_started":1}
I20260929 15:57:53.200314 29289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:53.193820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37537 (local address 127.23.222.65:43993)
0929 15:57:53.194903 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:53.194954 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:53.196625 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:53.197351 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:53.197359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:53.197456 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:53.198145 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:57:53.198147 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:53.198186 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:57:53.198269 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:53.198622 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:53.198632 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:53.198641 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 15:57:53.198944 (+ 303us) client_negotiation.cc:240] Negotiation successful
0929 15:57:53.199209 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":340,"threads_started":1}
I20260929 15:57:53.201879 29286 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37537
I20260929 15:57:53.202212 29286 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:53.202912 29286 heartbeater.cc:511] Master 127.23.222.126:37537 requested a full tablet report, sending...
I20260929 15:57:53.205345 29100 ts_manager.cc:194] Registered new tserver with Master: 769a3ffd5e64433e990b5608ba618aee (127.23.222.65:36743)
W20260929 15:57:53.531543 29290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:53.531839 29290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:53.532001 29290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:53.539598 29290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:53.539712 29290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:53.539891 29290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:57:53.549606 29290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:53.550935 29290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:53.552742 29290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:53.566437 29295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:53.566438 29296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:53.566497 29298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:53.567271 29290 server_base.cc:1034] running on GCE node
I20260929 15:57:53.567975 29290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:53.568985 29290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:53.570217 29290 hybrid_clock.cc:648] HybridClock initialized: now 1790697473570165 us; error 50 us; skew 500 ppm
I20260929 15:57:53.570560 29290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:57:53.573024 29290 webserver.cc:460] Webserver started at http://127.23.222.66:42573/ using document root <none> and password file <none>
I20260929 15:57:53.573774 29290 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:53.573909 29290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:53.574252 29290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:53.577337 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a"
format_stamp: "Formatted at 2026-09-29 15:57:53 on dist-test-slave-cq88"
server_key: "c5c02ad1a85d7d660a87d367a2fe133b"
server_key_iv: "18551f450089eef5a4ba9a2cb00f5077"
server_key_version: "encryptionkey@0"
I20260929 15:57:53.578207 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a"
format_stamp: "Formatted at 2026-09-29 15:57:53 on dist-test-slave-cq88"
server_key: "c5c02ad1a85d7d660a87d367a2fe133b"
server_key_iv: "18551f450089eef5a4ba9a2cb00f5077"
server_key_version: "encryptionkey@0"
I20260929 15:57:53.583695 29290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:57:53.587229 29304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:53.588855 29290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:53.589067 29290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a"
format_stamp: "Formatted at 2026-09-29 15:57:53 on dist-test-slave-cq88"
server_key: "c5c02ad1a85d7d660a87d367a2fe133b"
server_key_iv: "18551f450089eef5a4ba9a2cb00f5077"
server_key_version: "encryptionkey@0"
I20260929 15:57:53.589314 29290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:53.595587 29290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:53.596529 29290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:53.596927 29290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:53.597970 29290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:53.599547 29290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:53.599639 29290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:53.599802 29290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:53.599895 29290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:53.635524 29290 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:45199
I20260929 15:57:53.635613 29416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:45199 every 8 connection(s)
I20260929 15:57:53.636904 29290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:57:53.637948 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 29290
I20260929 15:57:53.638121 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:57:53.638721 24441 external_mini_cluster.cc:1428] Setting key efea00fb8277574c20adf94d88d43911
I20260929 15:57:53.642107 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37537
--builtin_ntp_servers=127.23.222.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 15:57:53.647336 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:53.640065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:56765 (local address 127.23.222.126:37537)
0929 15:57:53.640292 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:53.640298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:53.642040 (+ 1742us) server_negotiation.cc:407] Connection header received
0929 15:57:53.644896 (+ 2856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:53.644901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:53.644957 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:53.645038 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:53.646614 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:53.646619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:53.646622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:53.646639 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:53.646671 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:53.646683 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:53.646785 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:53.647001 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 15:57:53.647055 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:57:53.648739 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:53.640618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37537 (local address 127.23.222.66:56765)
0929 15:57:53.641857 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:53.641910 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:53.644536 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:53.645330 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:53.645345 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:53.645480 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:53.646227 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:57:53.646233 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:53.646288 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:57:53.646411 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:53.646782 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:53.646794 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:53.646805 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:57:53.647089 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 15:57:53.647398 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":344,"threads_started":1}
I20260929 15:57:53.650281 29417 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37537
I20260929 15:57:53.650599 29417 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:53.651218 29417 heartbeater.cc:511] Master 127.23.222.126:37537 requested a full tablet report, sending...
I20260929 15:57:53.652762 29100 ts_manager.cc:194] Registered new tserver with Master: e1f78b2924b44e8f9ee7f3ca6ca0c62a (127.23.222.66:45199)
W20260929 15:57:53.982600 29420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:53.982944 29420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:53.983114 29420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:53.990624 29420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:53.990775 29420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:53.990945 29420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:57:54.000856 29420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:54.002388 29420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:54.004311 29420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:54.018564 29425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:54.018648 29428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:54.018673 29426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:54.019312 29420 server_base.cc:1034] running on GCE node
I20260929 15:57:54.019922 29420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:54.020902 29420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:54.022111 29420 hybrid_clock.cc:648] HybridClock initialized: now 1790697474022070 us; error 41 us; skew 500 ppm
I20260929 15:57:54.022459 29420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:57:54.025008 29420 webserver.cc:460] Webserver started at http://127.23.222.67:34461/ using document root <none> and password file <none>
I20260929 15:57:54.025795 29420 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:54.025920 29420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:54.026276 29420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:54.029330 29420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "43563bbe46f549ba9e85815fd9e28f20"
format_stamp: "Formatted at 2026-09-29 15:57:54 on dist-test-slave-cq88"
server_key: "ab8406419e60339325d2b9c23f8edffd"
server_key_iv: "a0dfec59ee1f191d0e1183be61ddac91"
server_key_version: "encryptionkey@0"
I20260929 15:57:54.030170 29420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "43563bbe46f549ba9e85815fd9e28f20"
format_stamp: "Formatted at 2026-09-29 15:57:54 on dist-test-slave-cq88"
server_key: "ab8406419e60339325d2b9c23f8edffd"
server_key_iv: "a0dfec59ee1f191d0e1183be61ddac91"
server_key_version: "encryptionkey@0"
I20260929 15:57:54.035712 29420 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 15:57:54.039705 29434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:54.041154 29420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:54.041390 29420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "43563bbe46f549ba9e85815fd9e28f20"
format_stamp: "Formatted at 2026-09-29 15:57:54 on dist-test-slave-cq88"
server_key: "ab8406419e60339325d2b9c23f8edffd"
server_key_iv: "a0dfec59ee1f191d0e1183be61ddac91"
server_key_version: "encryptionkey@0"
I20260929 15:57:54.041677 29420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:54.048038 29420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:54.048942 29420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:54.049261 29420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:54.050274 29420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:57:54.051964 29420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:57:54.052096 29420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:54.052264 29420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:57:54.052363 29420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:54.086037 29420 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:45293
I20260929 15:57:54.086091 29546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:45293 every 8 connection(s)
I20260929 15:57:54.087572 29420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:57:54.095391 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 29420
I20260929 15:57:54.095571 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:57:54.095865 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.090143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:39041 (local address 127.23.222.126:37537)
0929 15:57:54.090345 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.090352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.091931 (+ 1579us) server_negotiation.cc:407] Connection header received
0929 15:57:54.093631 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.093638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.093712 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.093815 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.095084 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.095089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.095092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.095124 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.095155 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.095167 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.095306 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.095544 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.095622 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 15:57:54.096155 24441 external_mini_cluster.cc:1428] Setting key 81ae2c6bb44a19b90ff893e815a4f5d7
I20260929 15:57:54.097015 29549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.090732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37537 (local address 127.23.222.67:39041)
0929 15:57:54.091785 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:54.091826 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:54.093369 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:54.094008 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:54.094016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:54.094138 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:54.094758 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:57:54.094761 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:54.094797 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:57:54.094904 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:54.095291 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:54.095303 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:54.095311 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:57:54.095612 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 15:57:54.095892 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":263,"threads_started":1}
I20260929 15:57:54.098384 29547 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37537
I20260929 15:57:54.098690 29547 heartbeater.cc:464] Registering TS with master...
I20260929 15:57:54.099400 29547 heartbeater.cc:511] Master 127.23.222.126:37537 requested a full tablet report, sending...
I20260929 15:57:54.100672 29100 ts_manager.cc:194] Registered new tserver with Master: 43563bbe46f549ba9e85815fd9e28f20 (127.23.222.67:45293)
I20260929 15:57:54.112136 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:57:54.119526 29288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.116959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.23.222.126:37537)
0929 15:57:54.117193 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.117200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.117731 (+ 531us) server_negotiation.cc:407] Connection header received
0929 15:57:54.117880 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.117884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.117939 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.118027 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.118944 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.118948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.118952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.118967 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.118998 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.119010 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.119127 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.119207 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.119314 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:57:54.124140 29100 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:57:54.131021 29100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:57:54.151365 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.143890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36743 (local address 127.0.0.1:36076)
0929 15:57:54.145619 (+ 1729us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:54.145675 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:54.145802 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:54.146918 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:54.146926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:54.147494 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:54.148925 (+ 1431us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:57:54.148931 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:54.148976 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:57:54.149100 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:54.150348 (+ 1248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:54.150365 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:54.150379 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 15:57:54.150625 (+ 246us) client_negotiation.cc:240] Negotiation successful
0929 15:57:54.150769 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":2673,"threads_started":1}
I20260929 15:57:54.152297 29560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.144660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:45199 (local address 127.0.0.1:53940)
0929 15:57:54.146863 (+ 2203us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:54.146881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:54.146985 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:54.148626 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:54.148634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:54.148670 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:54.149796 (+ 1126us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:57:54.149802 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:54.149823 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:57:54.149881 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:54.150472 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:54.150480 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:54.150488 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 15:57:54.150679 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 15:57:54.151310 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2043,"mutex_wait_us":532,"thread_start_us":505,"threads_started":1}
I20260929 15:57:54.152467 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.143955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.23.222.65:36743)
0929 15:57:54.146353 (+ 2398us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.146363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.146395 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:57:54.146477 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.146488 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.146713 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.146894 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.149978 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.150000 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.150037 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.150064 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.150154 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.150190 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.151930 (+ 1740us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.152005 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.152132 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2217,"thread_start_us":2292,"threads_started":1}
I20260929 15:57:54.153275 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.144864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.23.222.66:45199)
0929 15:57:54.146451 (+ 1587us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.146459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.147110 (+ 651us) server_negotiation.cc:407] Connection header received
0929 15:57:54.147183 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.147192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.147474 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.147660 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.150098 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.150107 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.150126 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.150151 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.150227 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.150258 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.150412 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.150827 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.150960 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":1159,"threads_started":1}
I20260929 15:57:54.156744 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.147146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:45293 (local address 127.0.0.1:38216)
0929 15:57:54.151633 (+ 4487us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:54.151654 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:54.151735 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:54.154156 (+ 2421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:54.154161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:54.154200 (+ 39us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:54.155653 (+ 1453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:57:54.155657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:54.155676 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:57:54.155728 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:54.156301 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:54.156307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:54.156312 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 15:57:54.156398 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 15:57:54.156450 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4362,"thread_start_us":6279,"threads_started":1}
I20260929 15:57:54.157588 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.147174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.23.222.67:45293)
0929 15:57:54.153439 (+ 6265us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.153452 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.153498 (+ 46us) server_negotiation.cc:407] Connection header received
0929 15:57:54.153577 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.153588 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.153945 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.154140 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.156003 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.156014 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.156043 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.156067 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.156141 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.156167 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.157032 (+ 865us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.157117 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.157230 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6066,"thread_start_us":5894,"threads_started":1}
I20260929 15:57:54.157471 29221 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9f38f84900423db8cc015274bff055 (DEFAULT_TABLE table=test-table [id=621a1483b8c44ff3a683837ae7e315f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:54.157471 29352 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9f38f84900423db8cc015274bff055 (DEFAULT_TABLE table=test-table [id=621a1483b8c44ff3a683837ae7e315f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:54.158963 29221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9f38f84900423db8cc015274bff055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:54.158963 29352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9f38f84900423db8cc015274bff055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:54.161167 29482 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9f38f84900423db8cc015274bff055 (DEFAULT_TABLE table=test-table [id=621a1483b8c44ff3a683837ae7e315f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:57:54.162727 29482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9f38f84900423db8cc015274bff055. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:54.176975 29567 tablet_bootstrap.cc:492] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: Bootstrap starting.
I20260929 15:57:54.177243 29566 tablet_bootstrap.cc:492] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: Bootstrap starting.
I20260929 15:57:54.177019 29565 tablet_bootstrap.cc:492] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: Bootstrap starting.
I20260929 15:57:54.181306 29567 tablet_bootstrap.cc:654] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:54.182042 29566 tablet_bootstrap.cc:654] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:54.182696 29567 log.cc:824] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:54.183933 29566 log.cc:824] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:54.187829 29565 tablet_bootstrap.cc:654] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:54.191697 29565 log.cc:824] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:54.193538 29567 tablet_bootstrap.cc:492] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: No bootstrap required, opened a new log
I20260929 15:57:54.193857 29567 ts_tablet_manager.cc:1397] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20260929 15:57:54.194694 29566 tablet_bootstrap.cc:492] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: No bootstrap required, opened a new log
I20260929 15:57:54.194976 29566 ts_tablet_manager.cc:1397] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.010s
I20260929 15:57:54.201547 29565 tablet_bootstrap.cc:492] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: No bootstrap required, opened a new log
I20260929 15:57:54.201993 29565 ts_tablet_manager.cc:1397] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.005s
I20260929 15:57:54.204393 29567 raft_consensus.cc:348] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } }
I20260929 15:57:54.204227 29566 raft_consensus.cc:348] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } }
I20260929 15:57:54.204782 29567 raft_consensus.cc:374] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:54.204780 29566 raft_consensus.cc:374] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:54.204876 29566 raft_consensus.cc:729] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f78b2924b44e8f9ee7f3ca6ca0c62a, State: Initialized, Role: FOLLOWER
I20260929 15:57:54.204880 29567 raft_consensus.cc:729] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43563bbe46f549ba9e85815fd9e28f20, State: Initialized, Role: FOLLOWER
I20260929 15:57:54.205533 29567 consensus_queue.cc:260] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } }
I20260929 15:57:54.205650 29566 consensus_queue.cc:260] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } }
I20260929 15:57:54.206995 29417 heartbeater.cc:503] Master 127.23.222.126:37537 was elected leader, sending a full tablet report...
I20260929 15:57:54.207593 29566 ts_tablet_manager.cc:1428] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
I20260929 15:57:54.208617 29547 heartbeater.cc:503] Master 127.23.222.126:37537 was elected leader, sending a full tablet report...
I20260929 15:57:54.208978 29286 heartbeater.cc:503] Master 127.23.222.126:37537 was elected leader, sending a full tablet report...
I20260929 15:57:54.209350 29567 ts_tablet_manager.cc:1428] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260929 15:57:54.214393 29565 raft_consensus.cc:348] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } }
I20260929 15:57:54.214811 29565 raft_consensus.cc:374] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:54.214917 29565 raft_consensus.cc:729] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769a3ffd5e64433e990b5608ba618aee, State: Initialized, Role: FOLLOWER
I20260929 15:57:54.215564 29565 consensus_queue.cc:260] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } }
I20260929 15:57:54.217373 29565 ts_tablet_manager.cc:1428] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260929 15:57:54.299121 29573 raft_consensus.cc:482] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:57:54.299521 29573 raft_consensus.cc:504] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } }
I20260929 15:57:54.301568 29573 leader_election.cc:288] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43563bbe46f549ba9e85815fd9e28f20 (127.23.222.67:45293), e1f78b2924b44e8f9ee7f3ca6ca0c62a (127.23.222.66:45199)
I20260929 15:57:54.305034 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.301901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:37939 (local address 127.23.222.67:45293)
0929 15:57:54.302144 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.302154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.302950 (+ 796us) server_negotiation.cc:407] Connection header received
0929 15:57:54.303164 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.303170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.303230 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.303367 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.304389 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.304394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.304399 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.304416 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.304456 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.304468 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.304597 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.304689 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.304751 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:57:54.305163 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.302392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:38951 (local address 127.23.222.66:45199)
0929 15:57:54.302601 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.302608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.303220 (+ 612us) server_negotiation.cc:407] Connection header received
0929 15:57:54.303425 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.303429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.303478 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.303583 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.304551 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.304555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.304558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.304573 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.304608 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.304619 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.304738 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.304820 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.304884 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:57:54.305351 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.302027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:45199 (local address 127.23.222.65:38951)
0929 15:57:54.303108 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:54.303123 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:54.303273 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:54.303607 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:54.303611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:54.303638 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:54.304357 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:57:54.304359 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:54.304371 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:57:54.304404 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:54.304713 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:54.304718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:54.304722 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 15:57:54.304867 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 15:57:54.304916 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"thread_start_us":750,"threads_started":1}
I20260929 15:57:54.305490 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.301760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:45293 (local address 127.23.222.65:37939)
0929 15:57:54.302829 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:54.302863 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:54.303012 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:54.303406 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:54.303412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:54.303434 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:57:54.304183 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:57:54.304187 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:57:54.304208 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:57:54.304259 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:57:54.304560 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:57:54.304565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:57:54.304570 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 15:57:54.304718 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 15:57:54.304765 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":614,"threads_started":1}
I20260929 15:57:54.306387 29502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9f38f84900423db8cc015274bff055" candidate_uuid: "769a3ffd5e64433e990b5608ba618aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43563bbe46f549ba9e85815fd9e28f20" is_pre_election: true
I20260929 15:57:54.306385 29372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9f38f84900423db8cc015274bff055" candidate_uuid: "769a3ffd5e64433e990b5608ba618aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" is_pre_election: true
I20260929 15:57:54.306876 29372 raft_consensus.cc:2454] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769a3ffd5e64433e990b5608ba618aee in term 0.
I20260929 15:57:54.306883 29502 raft_consensus.cc:2454] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769a3ffd5e64433e990b5608ba618aee in term 0.
I20260929 15:57:54.307603 29175 leader_election.cc:302] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43563bbe46f549ba9e85815fd9e28f20, 769a3ffd5e64433e990b5608ba618aee, e1f78b2924b44e8f9ee7f3ca6ca0c62a; no voters:
I20260929 15:57:54.308007 29573 raft_consensus.cc:2781] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:57:54.308164 29573 raft_consensus.cc:482] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:57:54.308270 29573 raft_consensus.cc:3037] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:54.310387 29573 raft_consensus.cc:504] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } }
I20260929 15:57:54.310976 29573 leader_election.cc:288] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [CANDIDATE]: Term 1 election: Requested vote from peers 43563bbe46f549ba9e85815fd9e28f20 (127.23.222.67:45293), e1f78b2924b44e8f9ee7f3ca6ca0c62a (127.23.222.66:45199)
I20260929 15:57:54.311509 29372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9f38f84900423db8cc015274bff055" candidate_uuid: "769a3ffd5e64433e990b5608ba618aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a"
I20260929 15:57:54.311728 29502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9f38f84900423db8cc015274bff055" candidate_uuid: "769a3ffd5e64433e990b5608ba618aee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43563bbe46f549ba9e85815fd9e28f20"
I20260929 15:57:54.311764 29372 raft_consensus.cc:3037] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:54.311940 29502 raft_consensus.cc:3037] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:54.315279 29502 raft_consensus.cc:2454] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769a3ffd5e64433e990b5608ba618aee in term 1.
I20260929 15:57:54.315301 29372 raft_consensus.cc:2454] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769a3ffd5e64433e990b5608ba618aee in term 1.
I20260929 15:57:54.315773 29175 leader_election.cc:302] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769a3ffd5e64433e990b5608ba618aee, e1f78b2924b44e8f9ee7f3ca6ca0c62a; no voters:
I20260929 15:57:54.316120 29573 raft_consensus.cc:2781] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:54.316622 29573 raft_consensus.cc:686] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 1 LEADER]: Becoming Leader. State: Replica: 769a3ffd5e64433e990b5608ba618aee, State: Running, Role: LEADER
I20260929 15:57:54.317111 29573 consensus_queue.cc:237] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } }
I20260929 15:57:54.322367 29100 catalog_manager.cc:5495] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee reported cstate change: term changed from 0 to 1, leader changed from <none> to 769a3ffd5e64433e990b5608ba618aee (127.23.222.65). New cstate: current_term: 1 leader_uuid: "769a3ffd5e64433e990b5608ba618aee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "769a3ffd5e64433e990b5608ba618aee" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 } health_report { overall_health: UNKNOWN } } }
W20260929 15:57:54.339381 29548 tablet.cc:2369] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:57:54.348886 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.346566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.23.222.65:36743)
0929 15:57:54.346763 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.346768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.346789 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:57:54.346937 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.346945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.347051 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.347155 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.348281 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.348285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.348289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.348305 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.348337 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.348350 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.348453 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.348599 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.348647 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:57:54.358394 29372 raft_consensus.cc:1261] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [term 1 FOLLOWER]: Refusing update from remote peer 769a3ffd5e64433e990b5608ba618aee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:54.358394 29502 raft_consensus.cc:1261] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [term 1 FOLLOWER]: Refusing update from remote peer 769a3ffd5e64433e990b5608ba618aee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:57:54.359510 29576 consensus_queue.cc:1035] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f78b2924b44e8f9ee7f3ca6ca0c62a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 45199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:54.359929 29573 consensus_queue.cc:1035] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [LEADER]: Connected to new peer: Peer: permanent_uuid: "43563bbe46f549ba9e85815fd9e28f20" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:57:54.372612 29579 mvcc.cc:205] Tried to move back new op lower bound from 7334696854961557504 to 7334696854806794240. Current Snapshot: MvccSnapshot[applied={T|T < 7334696854961557504}]
I20260929 15:57:54.374545 29580 mvcc.cc:205] Tried to move back new op lower bound from 7334696854961557504 to 7334696854806794240. Current Snapshot: MvccSnapshot[applied={T|T < 7334696854961557504}]
I20260929 15:57:54.389022 29098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:54.389436 29098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:57:54.393610 29098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcd787bd69cb428fa696e8ecc5b511c1: Sending DeleteTablet for 3 replicas of tablet 8a9f38f84900423db8cc015274bff055
I20260929 15:57:54.394850 29221 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9f38f84900423db8cc015274bff055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:54 UTC) from {username='slave'} at 127.0.0.1:36076
I20260929 15:57:54.394932 29482 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9f38f84900423db8cc015274bff055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:54 UTC) from {username='slave'} at 127.0.0.1:38216
I20260929 15:57:54.395706 29352 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9f38f84900423db8cc015274bff055 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:57:54 UTC) from {username='slave'} at 127.0.0.1:53940
I20260929 15:57:54.395917 29589 tablet_replica.cc:330] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: stopping tablet replica
I20260929 15:57:54.396386 29589 raft_consensus.cc:2229] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:57:54.396646 29591 tablet_replica.cc:330] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: stopping tablet replica
I20260929 15:57:54.397073 29591 raft_consensus.cc:2229] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:54.397202 29589 raft_consensus.cc:2258] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:54.398150 29591 raft_consensus.cc:2258] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:54.399747 29589 ts_tablet_manager.cc:1905] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:54.400220 29591 ts_tablet_manager.cc:1905] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:54.402350 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 29159
I20260929 15:57:54.407353 29591 ts_tablet_manager.cc:1918] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:54.407619 29591 log.cc:1196] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/8a9f38f84900423db8cc015274bff055
I20260929 15:57:54.408238 29591 ts_tablet_manager.cc:1939] T 8a9f38f84900423db8cc015274bff055 P e1f78b2924b44e8f9ee7f3ca6ca0c62a: Deleting consensus metadata
I20260929 15:57:54.408214 29589 ts_tablet_manager.cc:1918] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:54.408390 29589 log.cc:1196] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/8a9f38f84900423db8cc015274bff055
I20260929 15:57:54.409009 29589 ts_tablet_manager.cc:1939] T 8a9f38f84900423db8cc015274bff055 P 769a3ffd5e64433e990b5608ba618aee: Deleting consensus metadata
I20260929 15:57:54.410410 29085 catalog_manager.cc:4841] TS e1f78b2924b44e8f9ee7f3ca6ca0c62a (127.23.222.66:45199): tablet 8a9f38f84900423db8cc015274bff055 (table test-table [id=621a1483b8c44ff3a683837ae7e315f2]) successfully deleted
I20260929 15:57:54.410789 29086 catalog_manager.cc:4841] TS 769a3ffd5e64433e990b5608ba618aee (127.23.222.65:36743): tablet 8a9f38f84900423db8cc015274bff055 (table test-table [id=621a1483b8c44ff3a683837ae7e315f2]) successfully deleted
I20260929 15:57:54.412576 29590 tablet_replica.cc:330] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: stopping tablet replica
I20260929 15:57:54.413133 29590 raft_consensus.cc:2229] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:57:54.413726 29590 raft_consensus.cc:2258] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:57:54.414230 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.403941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.23.222.65:36743)
0929 15:57:54.404137 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.404143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.411893 (+ 7750us) server_negotiation.cc:407] Connection header received
0929 15:57:54.412111 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.412121 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.412210 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.412320 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.413534 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.413541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.413546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.413565 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.413608 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.413621 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.413778 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.413861 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.413927 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:57:54.415459 29281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:54.416157 29590 ts_tablet_manager.cc:1905] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:57:54.422202 29590 ts_tablet_manager.cc:1918] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:57:54.422379 29590 log.cc:1196] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/8a9f38f84900423db8cc015274bff055
I20260929 15:57:54.422822 29590 ts_tablet_manager.cc:1939] T 8a9f38f84900423db8cc015274bff055 P 43563bbe46f549ba9e85815fd9e28f20: Deleting consensus metadata
I20260929 15:57:54.424517 29086 catalog_manager.cc:4841] TS 43563bbe46f549ba9e85815fd9e28f20 (127.23.222.67:45293): tablet 8a9f38f84900423db8cc015274bff055 (table test-table [id=621a1483b8c44ff3a683837ae7e315f2]) successfully deleted
I20260929 15:57:54.514932 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 29159
I20260929 15:57:54.533010 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 29290
I20260929 15:57:54.536191 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.533871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.23.222.66:45199)
0929 15:57:54.534100 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.534107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.534128 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:57:54.534341 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.534345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.534438 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.534553 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.535511 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.535517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.535522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.535546 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.535579 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.535591 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.535760 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.535826 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.535885 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 15:57:54.536984 29412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:54.632814 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 29290
I20260929 15:57:54.652905 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 29420
I20260929 15:57:54.655942 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:54.653836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.23.222.67:45293)
0929 15:57:54.654053 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:54.654057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:54.654075 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:57:54.654216 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:54.654220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:54.654299 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:54.654388 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:54.655335 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:54.655339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:54.655342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:54.655355 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:54.655385 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:54.655396 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:54.655504 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:54.655661 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 15:57:54.655709 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 15:57:54.656711 29542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:54.747571 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 29420
I20260929 15:57:54.766049 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 29069
I20260929 15:57:54.767387 29130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:54.875412 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 29069
2026-09-29T15:57:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2758 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 15:57:54.918565 24441 mini_oidc.cc:152] Starting JWKS server
I20260929 15:57:54.921334 24441 webserver.cc:460] Webserver started at https://127.0.0.1:40043/ using document root <none> and password file <none>
I20260929 15:57:54.921695 24441 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:57:54.922413 24441 webserver.cc:460] Webserver started at http://127.0.0.1:39763/ using document root <none> and password file <none>
2026-09-29T15:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:54Z Disabled control of system clock
I20260929 15:57:54.951349 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:33711
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:33711
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40043/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/testchainca.cert with env {}
W20260929 15:57:55.284265 29602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:55.284556 29602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:55.284644 29602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:55.291828 29602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:55.291934 29602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:55.291970 29602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:55.292025 29602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:55.292069 29602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:57:55.292124 29602 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:57:55.292156 29602 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40043/jwks.json
I20260929 15:57:55.302021 29602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:33711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:33711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40043/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:55.303278 29602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:55.304482 29602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:55.316635 29610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:55.316697 29607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:55.316727 29608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:55.317427 29602 server_base.cc:1034] running on GCE node
I20260929 15:57:55.318259 29602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:55.319782 29602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:55.321025 29602 hybrid_clock.cc:648] HybridClock initialized: now 1790697475320970 us; error 45 us; skew 500 ppm
I20260929 15:57:55.321410 29602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:57:55.323927 29602 webserver.cc:460] Webserver started at http://127.23.222.126:34229/ using document root <none> and password file <none>
I20260929 15:57:55.324625 29602 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:55.324761 29602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:55.325067 29602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:55.328078 29602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "2eeb820183fe4112be065ec290f2e23c"
format_stamp: "Formatted at 2026-09-29 15:57:55 on dist-test-slave-cq88"
server_key: "ecbfc3d7035d17cf4a405abcb9165f47"
server_key_iv: "de8f44c25b8a0cd656a9e199f09d9fd0"
server_key_version: "encryptionkey@0"
I20260929 15:57:55.328866 29602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "2eeb820183fe4112be065ec290f2e23c"
format_stamp: "Formatted at 2026-09-29 15:57:55 on dist-test-slave-cq88"
server_key: "ecbfc3d7035d17cf4a405abcb9165f47"
server_key_iv: "de8f44c25b8a0cd656a9e199f09d9fd0"
server_key_version: "encryptionkey@0"
I20260929 15:57:55.334051 29602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:55.337750 29616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:55.339277 29602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 15:57:55.339484 29602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "2eeb820183fe4112be065ec290f2e23c"
format_stamp: "Formatted at 2026-09-29 15:57:55 on dist-test-slave-cq88"
server_key: "ecbfc3d7035d17cf4a405abcb9165f47"
server_key_iv: "de8f44c25b8a0cd656a9e199f09d9fd0"
server_key_version: "encryptionkey@0"
I20260929 15:57:55.339725 29602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:55.356667 29602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:55.357553 29602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:55.357883 29602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:55.376448 29602 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:33711
I20260929 15:57:55.376454 29667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:33711 every 8 connection(s)
I20260929 15:57:55.377868 29602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:55.381861 29668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:55.381891 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 29602
I20260929 15:57:55.382136 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:55.382828 24441 external_mini_cluster.cc:1428] Setting key c695e9fd29773de5606a7096933c756d
I20260929 15:57:55.395954 29668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c: Bootstrap starting.
I20260929 15:57:55.397795 29671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.384935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.23.222.126:33711)
0929 15:57:55.386112 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:55.386124 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:55.386177 (+ 53us) server_negotiation.cc:407] Connection header received
0929 15:57:55.388217 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:55.388242 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:55.388772 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:55.389142 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:55.389823 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.391151 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.393161 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.393879 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.394023 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.395390 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:55.395400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:55.395415 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:55.395970 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:55.396058 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:55.396079 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:55.396223 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:55.396433 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 15:57:55.396770 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":463,"threads_started":1}
I20260929 15:57:55.399727 29668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:55.400930 29668 log.cc:824] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:55.401872 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:55.404215 29668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c: No bootstrap required, opened a new log
I20260929 15:57:55.411772 29668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eeb820183fe4112be065ec290f2e23c" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33711 } }
I20260929 15:57:55.412282 29668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:55.412407 29668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eeb820183fe4112be065ec290f2e23c, State: Initialized, Role: FOLLOWER
I20260929 15:57:55.412990 29668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eeb820183fe4112be065ec290f2e23c" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33711 } }
I20260929 15:57:55.413221 29668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:55.413360 29668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:55.413528 29668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:55.416006 29668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eeb820183fe4112be065ec290f2e23c" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33711 } }
I20260929 15:57:55.416536 29668 leader_election.cc:302] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eeb820183fe4112be065ec290f2e23c; no voters:
I20260929 15:57:55.417114 29668 leader_election.cc:288] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:55.417258 29673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:55.418404 29673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [term 1 LEADER]: Becoming Leader. State: Replica: 2eeb820183fe4112be065ec290f2e23c, State: Running, Role: LEADER
I20260929 15:57:55.418906 29673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eeb820183fe4112be065ec290f2e23c" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33711 } }
I20260929 15:57:55.419283 29668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:55.421867 29674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eeb820183fe4112be065ec290f2e23c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eeb820183fe4112be065ec290f2e23c" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33711 } } }
I20260929 15:57:55.422405 29674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:55.421838 29675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eeb820183fe4112be065ec290f2e23c. Latest consensus state: current_term: 1 leader_uuid: "2eeb820183fe4112be065ec290f2e23c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eeb820183fe4112be065ec290f2e23c" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33711 } } }
I20260929 15:57:55.422830 29675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:55.425462 29680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:55.432082 29680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:55.441172 29680 catalog_manager.cc:1347] Generated new cluster ID: ede0ce8032264ddc841109b195a66c8e
I20260929 15:57:55.441299 29680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:55.461843 29680 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:55.463200 29680 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:55.471967 29680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eeb820183fe4112be065ec290f2e23c: Generated new TSK 0
I20260929 15:57:55.472884 29680 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 15:57:55.518560 29698 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 15:57:55.518754 29698 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:57:55.520025 29671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:55.497462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.23.222.126:33711)
0929 15:57:55.497689 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:55.497697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:55.498226 (+ 529us) server_negotiation.cc:407] Connection header received
0929 15:57:55.498520 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:55.498526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:55.498633 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:55.498807 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:57:55.499327 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.500493 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.501955 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.502611 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.502742 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.503383 (+ 641us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:57:55.519293 (+ 15910us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 15:57:55.519671 (+ 378us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36868: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 15:57:55.520169 29671 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36868: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 15:57:55.521723 29697 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:55.497340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:33711 (local address 127.0.0.1:36868)
0929 15:57:55.498118 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:55.498136 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:55.498295 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:55.498829 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:55.498832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:55.498842 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:57:55.499152 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:55.499160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.500644 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.500649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:55.501806 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:55.501818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:55.502820 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:55.502825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:55.502956 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:55.503031 (+ 75us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:57:55.519599 (+ 16568us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 15:57:55.519921 (+ 322us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.222.126:33711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":398,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.222.126:33711: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40043/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 15:57:55.524055 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 29602
I20260929 15:57:55.525273 29663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:55.621140 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 29602
2026-09-29T15:57:55Z chronyd exiting
I20260929 15:57:55.672209 24441 test_util.cc:175] -----------------------------------------------
I20260929 15:57:55.672359 24441 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790697431711191-24441-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (758 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 15:57:55.676158 24441 mini_oidc.cc:152] Starting JWKS server
I20260929 15:57:55.677953 24441 webserver.cc:460] Webserver started at https://127.0.0.1:35355/ using document root <none> and password file <none>
I20260929 15:57:55.678159 24441 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:57:55.678733 24441 webserver.cc:460] Webserver started at http://127.0.0.1:33039/ using document root <none> and password file <none>
2026-09-29T15:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:55Z Disabled control of system clock
I20260929 15:57:55.705356 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:36931
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:37473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:36931
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35355/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/testchainca.cert with env {}
W20260929 15:57:56.054949 29705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:56.055349 29705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:56.055508 29705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:56.062958 29705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:56.063133 29705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:56.063227 29705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:56.063346 29705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:56.063426 29705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:57:56.063517 29705 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:57:56.063601 29705 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35355/jwks.json
I20260929 15:57:56.073896 29705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:36931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:36931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35355/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:56.075423 29705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:56.076783 29705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:56.089766 29710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:56.089740 29711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:56.089793 29713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:56.090700 29705 server_base.cc:1034] running on GCE node
I20260929 15:57:56.091655 29705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:56.093192 29705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:56.094480 29705 hybrid_clock.cc:648] HybridClock initialized: now 1790697476094430 us; error 55 us; skew 500 ppm
I20260929 15:57:56.094882 29705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:57:56.097393 29705 webserver.cc:460] Webserver started at http://127.23.222.126:41567/ using document root <none> and password file <none>
I20260929 15:57:56.098176 29705 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:56.098318 29705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:56.098661 29705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:56.101800 29705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "8a55050934e149c18e048ebbe820f476"
format_stamp: "Formatted at 2026-09-29 15:57:56 on dist-test-slave-cq88"
server_key: "eff3e7d0b41b144e25705c92074bac95"
server_key_iv: "98267cf6f894feab91716b186acf3937"
server_key_version: "encryptionkey@0"
I20260929 15:57:56.102644 29705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "8a55050934e149c18e048ebbe820f476"
format_stamp: "Formatted at 2026-09-29 15:57:56 on dist-test-slave-cq88"
server_key: "eff3e7d0b41b144e25705c92074bac95"
server_key_iv: "98267cf6f894feab91716b186acf3937"
server_key_version: "encryptionkey@0"
I20260929 15:57:56.108471 29705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 15:57:56.112561 29719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:56.114244 29705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:57:56.114485 29705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "8a55050934e149c18e048ebbe820f476"
format_stamp: "Formatted at 2026-09-29 15:57:56 on dist-test-slave-cq88"
server_key: "eff3e7d0b41b144e25705c92074bac95"
server_key_iv: "98267cf6f894feab91716b186acf3937"
server_key_version: "encryptionkey@0"
I20260929 15:57:56.114741 29705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:56.133489 29705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:56.134472 29705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:56.134816 29705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:56.153941 29705 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:36931
I20260929 15:57:56.153964 29770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:36931 every 8 connection(s)
I20260929 15:57:56.155428 29705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:56.156421 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 29705
I20260929 15:57:56.156625 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:56.157287 24441 external_mini_cluster.cc:1428] Setting key c5d9cdfa9e313e640f5a76b82d6186bf
I20260929 15:57:56.160104 29771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:56.171868 29771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476: Bootstrap starting.
I20260929 15:57:56.171977 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:56.159297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.23.222.126:36931)
0929 15:57:56.160420 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:56.160434 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:56.160481 (+ 47us) server_negotiation.cc:407] Connection header received
0929 15:57:56.162194 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:56.162217 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:56.162776 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:56.163245 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:56.163906 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:56.165431 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:56.166913 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:56.167772 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:56.168026 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:56.169408 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:56.169416 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:56.169433 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:56.170041 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:56.170131 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:56.170154 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:56.170296 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:56.170534 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 15:57:56.170883 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":369,"threads_started":1}
I20260929 15:57:56.175673 29771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:56.176198 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:56.176821 29771 log.cc:824] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:56.180605 29771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476: No bootstrap required, opened a new log
I20260929 15:57:56.190058 29771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55050934e149c18e048ebbe820f476" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36931 } }
I20260929 15:57:56.190655 29771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:56.190804 29771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a55050934e149c18e048ebbe820f476, State: Initialized, Role: FOLLOWER
I20260929 15:57:56.191530 29771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55050934e149c18e048ebbe820f476" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36931 } }
I20260929 15:57:56.191800 29771 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:56.191946 29771 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:56.192138 29771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:56.194376 29771 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55050934e149c18e048ebbe820f476" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36931 } }
I20260929 15:57:56.194944 29771 leader_election.cc:302] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a55050934e149c18e048ebbe820f476; no voters:
I20260929 15:57:56.195674 29771 leader_election.cc:288] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:56.195798 29776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:56.196534 29776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [term 1 LEADER]: Becoming Leader. State: Replica: 8a55050934e149c18e048ebbe820f476, State: Running, Role: LEADER
I20260929 15:57:56.197175 29776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55050934e149c18e048ebbe820f476" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36931 } }
I20260929 15:57:56.198071 29771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:56.200507 29777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a55050934e149c18e048ebbe820f476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55050934e149c18e048ebbe820f476" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36931 } } }
I20260929 15:57:56.200982 29777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:56.207695 29778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a55050934e149c18e048ebbe820f476. Latest consensus state: current_term: 1 leader_uuid: "8a55050934e149c18e048ebbe820f476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a55050934e149c18e048ebbe820f476" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36931 } } }
I20260929 15:57:56.208130 29778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:56.208362 29785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:56.213342 29785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:56.222558 29785 catalog_manager.cc:1347] Generated new cluster ID: 95bb05be61cc4d2d8f928bc6358e4402
I20260929 15:57:56.222692 29785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:56.237663 29785 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:56.239635 29785 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:56.252851 29785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a55050934e149c18e048ebbe820f476: Generated new TSK 0
I20260929 15:57:56.254092 29785 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 15:57:56.299191 29801 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 15:57:56.299373 29801 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:57:56.300664 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:56.279140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36450 (local address 127.23.222.126:36931)
0929 15:57:56.279445 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:56.279455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:56.279984 (+ 529us) server_negotiation.cc:407] Connection header received
0929 15:57:56.280221 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:56.280227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:56.280340 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:56.280518 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:57:56.281004 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:56.282238 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:56.284121 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:56.284857 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:56.285012 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:56.285588 (+ 576us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:57:56.299917 (+ 14329us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 15:57:56.300318 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 15:57:56.300704 29800 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:56.278976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:36931 (local address 127.0.0.1:36450)
0929 15:57:56.279878 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:56.279894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:56.280019 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:56.280554 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:56.280558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:56.280571 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:57:56.280823 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:56.280831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:56.282405 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:56.282409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:56.283970 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:56.283983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:56.285086 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:56.285091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:56.285224 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:56.285259 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:57:56.300207 (+ 14948us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 15:57:56.300419 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.222.126:36931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":631,"threads_started":1}
W20260929 15:57:56.300793 29774 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260929 15:57:56.301819 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 29705
I20260929 15:57:56.303047 29766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:56.398442 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 29705
2026-09-29T15:57:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (778 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 15:57:56.454274 24441 mini_oidc.cc:152] Starting JWKS server
I20260929 15:57:56.456171 24441 webserver.cc:460] Webserver started at https://127.0.0.1:46013/ using document root <none> and password file <none>
I20260929 15:57:56.456398 24441 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 15:57:56.457036 24441 webserver.cc:460] Webserver started at http://127.0.0.1:41125/ using document root <none> and password file <none>
2026-09-29T15:57:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:56Z Disabled control of system clock
I20260929 15:57:56.484323 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38167
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:34779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:38167
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46013/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 15:57:56.814023 29808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:56.814347 29808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:56.814491 29808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:56.821893 29808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:56.822023 29808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:56.822098 29808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:56.822201 29808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:56.822271 29808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:57:56.822355 29808 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:57:56.822436 29808 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46013/jwks.json
I20260929 15:57:56.832319 29808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:34779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:38167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46013/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:56.834185 29808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:56.836488 29808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:56.849249 29816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:56.849341 29814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:56.849316 29813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:56.850063 29808 server_base.cc:1034] running on GCE node
I20260929 15:57:56.850979 29808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:56.852614 29808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:56.853826 29808 hybrid_clock.cc:648] HybridClock initialized: now 1790697476853794 us; error 38 us; skew 500 ppm
I20260929 15:57:56.854163 29808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:57:56.856600 29808 webserver.cc:460] Webserver started at http://127.23.222.126:39547/ using document root <none> and password file <none>
I20260929 15:57:56.857296 29808 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:56.857386 29808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:56.857733 29808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:56.860818 29808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "43c77394ee7f4ac190ac1c2288698060"
format_stamp: "Formatted at 2026-09-29 15:57:56 on dist-test-slave-cq88"
server_key: "b4d330c4e685a2e355bc0534a4b341fa"
server_key_iv: "6bfc6aafb20e276b4139d23e638c1b95"
server_key_version: "encryptionkey@0"
I20260929 15:57:56.861579 29808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "43c77394ee7f4ac190ac1c2288698060"
format_stamp: "Formatted at 2026-09-29 15:57:56 on dist-test-slave-cq88"
server_key: "b4d330c4e685a2e355bc0534a4b341fa"
server_key_iv: "6bfc6aafb20e276b4139d23e638c1b95"
server_key_version: "encryptionkey@0"
I20260929 15:57:56.866976 29808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:57:56.870600 29822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:56.872188 29808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:57:56.872345 29808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "43c77394ee7f4ac190ac1c2288698060"
format_stamp: "Formatted at 2026-09-29 15:57:56 on dist-test-slave-cq88"
server_key: "b4d330c4e685a2e355bc0534a4b341fa"
server_key_iv: "6bfc6aafb20e276b4139d23e638c1b95"
server_key_version: "encryptionkey@0"
I20260929 15:57:56.872567 29808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:56.911845 29808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:56.912711 29808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:56.912999 29808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:56.931807 29808 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:38167
I20260929 15:57:56.931849 29873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:38167 every 8 connection(s)
I20260929 15:57:56.933205 29808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:56.935194 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 29808
I20260929 15:57:56.935505 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:56.936148 24441 external_mini_cluster.cc:1428] Setting key 9ef91aeeccaf88c97f962f1e8e996bd0
I20260929 15:57:56.937661 29874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:56.950295 29874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060: Bootstrap starting.
I20260929 15:57:56.950577 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:56.938046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.23.222.126:38167)
0929 15:57:56.939659 (+ 1613us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:56.939670 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:56.939710 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:57:56.941266 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:56.941287 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:56.941790 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:56.942212 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:56.942775 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:56.944215 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:56.945622 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:56.946306 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:56.946501 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:56.947883 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:56.947892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:56.947908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:56.948594 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:56.948698 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:56.948722 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:56.948865 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:56.949132 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 15:57:56.949463 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":367,"threads_started":1}
I20260929 15:57:56.953775 29874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:56.954865 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:56.954941 29874 log.cc:824] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:56.957962 29874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060: No bootstrap required, opened a new log
I20260929 15:57:56.965598 29874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c77394ee7f4ac190ac1c2288698060" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38167 } }
I20260929 15:57:56.965996 29874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:56.966117 29874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c77394ee7f4ac190ac1c2288698060, State: Initialized, Role: FOLLOWER
I20260929 15:57:56.966742 29874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c77394ee7f4ac190ac1c2288698060" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38167 } }
I20260929 15:57:56.966982 29874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:56.967139 29874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:56.967332 29874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:56.969455 29874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c77394ee7f4ac190ac1c2288698060" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38167 } }
I20260929 15:57:56.970021 29874 leader_election.cc:302] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43c77394ee7f4ac190ac1c2288698060; no voters:
I20260929 15:57:56.970664 29874 leader_election.cc:288] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:56.970815 29879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:56.972301 29879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [term 1 LEADER]: Becoming Leader. State: Replica: 43c77394ee7f4ac190ac1c2288698060, State: Running, Role: LEADER
I20260929 15:57:56.972817 29879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c77394ee7f4ac190ac1c2288698060" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38167 } }
I20260929 15:57:56.972965 29874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:56.975926 29881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43c77394ee7f4ac190ac1c2288698060. Latest consensus state: current_term: 1 leader_uuid: "43c77394ee7f4ac190ac1c2288698060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c77394ee7f4ac190ac1c2288698060" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38167 } } }
I20260929 15:57:56.975956 29880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43c77394ee7f4ac190ac1c2288698060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43c77394ee7f4ac190ac1c2288698060" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38167 } } }
I20260929 15:57:56.976356 29881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:56.976390 29880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:56.979059 29888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:56.983668 29888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:56.992812 29888 catalog_manager.cc:1347] Generated new cluster ID: 4f83cc01c4a94704bdd451d9e0e1f5e1
I20260929 15:57:56.992938 29888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:57.029755 29888 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:57.031454 29888 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:57.037778 29888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43c77394ee7f4ac190ac1c2288698060: Generated new TSK 0
I20260929 15:57:57.038933 29888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:57.069317 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:57.060353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43916 (local address 127.23.222.126:38167)
0929 15:57:57.060602 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:57.060612 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:57.061145 (+ 533us) server_negotiation.cc:407] Connection header received
0929 15:57:57.061397 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:57.061401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:57.061478 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:57.061626 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:57:57.062200 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.062933 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.065014 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.065480 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.065582 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:57.066221 (+ 639us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:57:57.068675 (+ 2454us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 15:57:57.069000 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 15:57:57.069451 29877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43916: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 15:57:57.071570 29903 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:57.060318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38167 (local address 127.0.0.1:43916)
0929 15:57:57.061016 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:57.061036 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:57.061197 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:57.061661 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:57.061667 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:57.061682 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:57:57.062045 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:57.062058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.063078 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.063084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:57.064843 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:57.064860 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.065679 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.065686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:57.065884 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:57.065935 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:57:57.068900 (+ 2965us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 15:57:57.071058 (+ 2158us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.222.126:38167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":1807,"thread_start_us":357,"threads_started":1}
I20260929 15:57:57.072685 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 29808
I20260929 15:57:57.074012 29869 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 15:57:57.081650 29904 webserver.cc:533] Webserver: sslize failed:
E20260929 15:57:57.081771 29904 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 15:57:57.166445 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 29808
2026-09-29T15:57:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (770 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T15:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:57:57Z Disabled control of system clock
I20260929 15:57:57.251855 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:45651
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:45651
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/testchainca.cert with env {}
W20260929 15:57:57.583182 29909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:57:57.583508 29909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:57:57.583659 29909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:57:57.591003 29909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:57:57.591125 29909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:57:57.591198 29909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:57:57.591332 29909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:57:57.591398 29909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 15:57:57.591486 29909 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 15:57:57.591542 29909 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 15:57:57.601812 29909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:45651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:45651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:57:57.603220 29909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:57:57.604633 29909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:57:57.617017 29915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:57.617084 29917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:57:57.617084 29914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:57:57.618237 29909 server_base.cc:1034] running on GCE node
I20260929 15:57:57.619055 29909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:57:57.620612 29909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:57:57.621814 29909 hybrid_clock.cc:648] HybridClock initialized: now 1790697477621763 us; error 44 us; skew 500 ppm
I20260929 15:57:57.622200 29909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:57:57.632813 29909 webserver.cc:460] Webserver started at http://127.23.222.126:44409/ using document root <none> and password file <none>
I20260929 15:57:57.633630 29909 fs_manager.cc:359] Metadata directory not provided
I20260929 15:57:57.633770 29909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:57:57.634130 29909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:57:57.637291 29909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "62d4f5725601444d937a0b9624c1e953"
format_stamp: "Formatted at 2026-09-29 15:57:57 on dist-test-slave-cq88"
server_key: "c106224b4dc491ddae9e51233ab34954"
server_key_iv: "fef104ab561c7ff3187f5cba978c1df4"
server_key_version: "encryptionkey@0"
I20260929 15:57:57.638202 29909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "62d4f5725601444d937a0b9624c1e953"
format_stamp: "Formatted at 2026-09-29 15:57:57 on dist-test-slave-cq88"
server_key: "c106224b4dc491ddae9e51233ab34954"
server_key_iv: "fef104ab561c7ff3187f5cba978c1df4"
server_key_version: "encryptionkey@0"
I20260929 15:57:57.643769 29909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:57:57.647612 29923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:57:57.649205 29909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:57:57.649434 29909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "62d4f5725601444d937a0b9624c1e953"
format_stamp: "Formatted at 2026-09-29 15:57:57 on dist-test-slave-cq88"
server_key: "c106224b4dc491ddae9e51233ab34954"
server_key_iv: "fef104ab561c7ff3187f5cba978c1df4"
server_key_version: "encryptionkey@0"
I20260929 15:57:57.649677 29909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:57:57.686520 29909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:57:57.687523 29909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:57:57.687855 29909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:57:57.708078 29909 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:45651
I20260929 15:57:57.708103 29974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:45651 every 8 connection(s)
I20260929 15:57:57.709512 29909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:57:57.712668 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 29909
I20260929 15:57:57.712874 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:57:57.713693 24441 external_mini_cluster.cc:1428] Setting key eb2c086167eebbf784b47b091099637e
I20260929 15:57:57.714205 29975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:57:57.727672 29975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953: Bootstrap starting.
I20260929 15:57:57.728581 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:57.715824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.23.222.126:45651)
0929 15:57:57.716884 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:57.716896 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:57.716941 (+ 45us) server_negotiation.cc:407] Connection header received
0929 15:57:57.719183 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:57.719209 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:57.719838 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:57.720306 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:57:57.720947 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.722258 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.723636 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.724350 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.724595 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:57.725875 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:57:57.725883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:57:57.725899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:57:57.726452 (+ 553us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:57:57.726543 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:57:57.726566 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:57:57.726712 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 15:57:57.726938 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 15:57:57.727341 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":379,"threads_started":1}
I20260929 15:57:57.731060 29975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953: Neither blocks nor log segments found. Creating new log.
I20260929 15:57:57.732231 29975 log.cc:824] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953: Log is configured to *not* fsync() on all Append() calls
I20260929 15:57:57.732820 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 15:57:57.735870 29975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953: No bootstrap required, opened a new log
I20260929 15:57:57.743546 29975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4f5725601444d937a0b9624c1e953" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45651 } }
I20260929 15:57:57.743963 29975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:57:57.744079 29975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62d4f5725601444d937a0b9624c1e953, State: Initialized, Role: FOLLOWER
I20260929 15:57:57.744691 29975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4f5725601444d937a0b9624c1e953" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45651 } }
I20260929 15:57:57.744932 29975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:57:57.745098 29975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:57:57.745275 29975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:57:57.747650 29975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4f5725601444d937a0b9624c1e953" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45651 } }
I20260929 15:57:57.748238 29975 leader_election.cc:302] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62d4f5725601444d937a0b9624c1e953; no voters:
I20260929 15:57:57.748934 29975 leader_election.cc:288] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:57:57.749049 29980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:57:57.749737 29980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [term 1 LEADER]: Becoming Leader. State: Replica: 62d4f5725601444d937a0b9624c1e953, State: Running, Role: LEADER
I20260929 15:57:57.750293 29980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4f5725601444d937a0b9624c1e953" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45651 } }
I20260929 15:57:57.751386 29975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:57:57.753273 29981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62d4f5725601444d937a0b9624c1e953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4f5725601444d937a0b9624c1e953" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45651 } } }
I20260929 15:57:57.753737 29981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:57.753756 29982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62d4f5725601444d937a0b9624c1e953. Latest consensus state: current_term: 1 leader_uuid: "62d4f5725601444d937a0b9624c1e953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d4f5725601444d937a0b9624c1e953" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45651 } } }
I20260929 15:57:57.755928 29982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953 [sys.catalog]: This master's current role is: LEADER
I20260929 15:57:57.755805 29987 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:57:57.762069 29987 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:57:57.771927 29987 catalog_manager.cc:1347] Generated new cluster ID: cfaf3773f3aa4590989aa975deea7268
I20260929 15:57:57.772086 29987 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:57:57.794035 29987 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:57:57.795544 29987 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:57:57.810431 29987 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62d4f5725601444d937a0b9624c1e953: Generated new TSK 0
I20260929 15:57:57.811643 29987 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:57:58.289765 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:57:57.837416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.23.222.126:45651)
0929 15:57:57.838578 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0929 15:57:57.838586 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:57:57.838613 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:57:57.838758 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:57:57.838767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:57:57.838866 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:57:57.839062 (+ 196us) server_negotiation.cc:226] Negotiated authn=JWT
0929 15:57:57.839712 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.840868 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.842299 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.843073 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.843206 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:57.843821 (+ 615us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 15:57:58.289207 (+445386us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 15:57:58.289540 (+ 333us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33802: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":989}
W20260929 15:57:58.289870 29978 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33802: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 15:57:58.290200 30004 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:57:57.837257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:45651 (local address 127.0.0.1:33802)
0929 15:57:57.837966 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 15:57:57.838015 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 15:57:57.838219 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:57:57.839275 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:57:57.839282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:57:57.839295 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 15:57:57.839540 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:57.839550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.841052 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.841056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:57.842145 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:57:57.842156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:57:57.843311 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:57:57.843322 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:57:57.843460 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:57:57.843498 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 15:57:58.289548 (+446050us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 15:57:58.289804 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.222.126:45651: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":393,"threads_started":1}
I20260929 15:57:58.291385 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 29909
I20260929 15:57:58.292523 29970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:57:58.396315 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 29909
2026-09-29T15:57:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1209 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (423 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1513 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:00 dist-test-slave-cq88 krb5kdc[30020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:00 dist-test-slave-cq88 krb5kdc[30020](info): set up 2 sockets
Sep 29 15:58:00 dist-test-slave-cq88 krb5kdc[30020](info): commencing operation
krb5kdc: starting...
W20260929 15:58:01.434909 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:58:01 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:58:01.655951 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:43581
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:43581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:01.990813 30033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:01.991142 30033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:01.991290 30033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:01.998805 30033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:01.998936 30033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:01.998982 30033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:01.999045 30033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:01.999091 30033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:02.009653 30033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:43581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:43581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:02.011088 30033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:02.012766 30033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:02.026464 30038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:02.026803 30039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:02.027901 30041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:02.028363 30033 server_base.cc:1034] running on GCE node
I20260929 15:58:02.029270 30033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:02.030804 30033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:02.032044 30033 hybrid_clock.cc:648] HybridClock initialized: now 1790697482031975 us; error 69 us; skew 500 ppm
I20260929 15:58:02.032487 30033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:02 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697482, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:02.036190 30033 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:58:02.037989 30033 webserver.cc:460] Webserver started at http://127.23.222.126:33721/ using document root <none> and password file <none>
I20260929 15:58:02.038772 30033 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:02.038937 30033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:02.039361 30033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:02.042637 30033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50"
format_stamp: "Formatted at 2026-09-29 15:58:02 on dist-test-slave-cq88"
server_key: "9bad9aee1a9329ca62376c9a8b3132bc"
server_key_iv: "540dbc1988d27da38596a1cb64b81bf8"
server_key_version: "encryptionkey@0"
I20260929 15:58:02.043908 30033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50"
format_stamp: "Formatted at 2026-09-29 15:58:02 on dist-test-slave-cq88"
server_key: "9bad9aee1a9329ca62376c9a8b3132bc"
server_key_iv: "540dbc1988d27da38596a1cb64b81bf8"
server_key_version: "encryptionkey@0"
I20260929 15:58:02.049872 30033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:02.054005 30048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:02.055686 30033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:58:02.055923 30033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50"
format_stamp: "Formatted at 2026-09-29 15:58:02 on dist-test-slave-cq88"
server_key: "9bad9aee1a9329ca62376c9a8b3132bc"
server_key_iv: "540dbc1988d27da38596a1cb64b81bf8"
server_key_version: "encryptionkey@0"
I20260929 15:58:02.056161 30033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:02.081270 30033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:02.086001 30033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:02.086308 30033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:02.106156 30033 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:43581
I20260929 15:58:02.106254 30099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:43581 every 8 connection(s)
I20260929 15:58:02.107616 30033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:02.111770 30100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:02.118232 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 30033
I20260929 15:58:02.118485 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:58:02.119449 24441 external_mini_cluster.cc:1428] Setting key b187b0c430b903e0481d46b0a11b1896
I20260929 15:58:02.125133 30100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50: Bootstrap starting.
I20260929 15:58:02.128645 30100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:02.129752 30100 log.cc:824] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:02.132943 30100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50: No bootstrap required, opened a new log
Sep 29 15:58:02 dist-test-slave-cq88 krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:02.142182 30100 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 43581 } }
I20260929 15:58:02.142643 30100 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:02.142800 30100 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd7dc0bdb0e6456bb85e41aa9b230a50, State: Initialized, Role: FOLLOWER
I20260929 15:58:02.143559 30100 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 43581 } }
I20260929 15:58:02.143858 30100 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:02.144045 30100 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:02.144256 30100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:02.144295 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:02.122129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56714 (local address 127.23.222.126:43581)
0929 15:58:02.123018 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:02.123032 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:02.123077 (+ 45us) server_negotiation.cc:407] Connection header received
0929 15:58:02.124940 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:02.124964 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:02.125474 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:02.125907 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:02.127466 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.128819 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.130128 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.131360 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.131561 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:02.136119 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:02.136129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:02.136146 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:02.136191 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:02.139640 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:02.140452 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:02.140460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:02.140472 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:02.140681 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:02.141103 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:02.141108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:02.141111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:02.141723 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:02.142030 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:02.142503 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 15:58:02.143052 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":383,"threads_started":1}
I20260929 15:58:02.147380 30100 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 43581 } }
I20260929 15:58:02.148054 30100 leader_election.cc:302] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd7dc0bdb0e6456bb85e41aa9b230a50; no voters:
I20260929 15:58:02.148710 30100 leader_election.cc:288] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:02.148864 30105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:02.149624 30105 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [term 1 LEADER]: Becoming Leader. State: Replica: fd7dc0bdb0e6456bb85e41aa9b230a50, State: Running, Role: LEADER
I20260929 15:58:02.150220 30105 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 43581 } }
I20260929 15:58:02.151063 30100 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:02.153342 30106 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 43581 } } }
I20260929 15:58:02.153995 30106 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:02.153626 30107 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd7dc0bdb0e6456bb85e41aa9b230a50. Latest consensus state: current_term: 1 leader_uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd7dc0bdb0e6456bb85e41aa9b230a50" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 43581 } } }
I20260929 15:58:02.155001 30107 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:02.155850 30111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:02.161325 30111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:02.171206 30111 catalog_manager.cc:1347] Generated new cluster ID: 56236df563f94e12ae60b4a166891d78
I20260929 15:58:02.171375 30111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:02.184697 30111 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:02.186614 30111 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:02.198310 30111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd7dc0bdb0e6456bb85e41aa9b230a50: Generated new TSK 0
I20260929 15:58:02.199220 30111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:58:02.260780 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:43581
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:02.601075 30126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:02.601454 30126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:02.601632 30126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:02.609069 30126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:02.609239 30126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:02.609417 30126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:58:02.619455 30126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:43581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:02.620924 30126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:02.622485 30126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:02.636917 30131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:02.636891 30134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:02.636873 30132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:02.637792 30126 server_base.cc:1034] running on GCE node
I20260929 15:58:02.638450 30126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:02.639582 30126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:02.640820 30126 hybrid_clock.cc:648] HybridClock initialized: now 1790697482640743 us; error 78 us; skew 500 ppm
I20260929 15:58:02.641158 30126 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:02 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697482, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:02.648916 30126 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:58:02.650636 30126 webserver.cc:460] Webserver started at http://127.23.222.65:37745/ using document root <none> and password file <none>
I20260929 15:58:02.651500 30126 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:02.651649 30126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:02.652012 30126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:02.655077 30126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "aa9587efc4ef42a298d968ccb568befb"
format_stamp: "Formatted at 2026-09-29 15:58:02 on dist-test-slave-cq88"
server_key: "d00969faa5645284d1a5f05e97113757"
server_key_iv: "b8fc0e08e1d86d3cb316ee18e72a2b3a"
server_key_version: "encryptionkey@0"
I20260929 15:58:02.656100 30126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "aa9587efc4ef42a298d968ccb568befb"
format_stamp: "Formatted at 2026-09-29 15:58:02 on dist-test-slave-cq88"
server_key: "d00969faa5645284d1a5f05e97113757"
server_key_iv: "b8fc0e08e1d86d3cb316ee18e72a2b3a"
server_key_version: "encryptionkey@0"
I20260929 15:58:02.661811 30126 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:02.665431 30141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:02.666759 30126 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:02.666936 30126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "aa9587efc4ef42a298d968ccb568befb"
format_stamp: "Formatted at 2026-09-29 15:58:02 on dist-test-slave-cq88"
server_key: "d00969faa5645284d1a5f05e97113757"
server_key_iv: "b8fc0e08e1d86d3cb316ee18e72a2b3a"
server_key_version: "encryptionkey@0"
I20260929 15:58:02.667150 30126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:02.678402 30126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:02.682853 30126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:02.683163 30126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:02.684211 30126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:02.685580 30126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:02.685689 30126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:02.685838 30126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:02.685912 30126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:02.721910 30126 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:44151
I20260929 15:58:02.721952 30253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:44151 every 8 connection(s)
I20260929 15:58:02.723407 30126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:02.724769 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 30126
I20260929 15:58:02.724996 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:02.725924 24441 external_mini_cluster.cc:1428] Setting key fa2343d08f4e78aefb8fda74bd3b1d7d
Sep 29 15:58:02 dist-test-slave-cq88 krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697482, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 15:58:02.748935 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:02.726925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:54535 (local address 127.23.222.126:43581)
0929 15:58:02.727448 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:02.727454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:02.728649 (+ 1195us) server_negotiation.cc:407] Connection header received
0929 15:58:02.730718 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:02.730724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:02.730838 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:02.730979 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:02.732790 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.733955 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.735776 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.736648 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.736814 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:02.740937 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:02.740948 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:02.740956 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:02.741008 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:02.744999 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:02.745945 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:02.745954 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:02.745958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:02.746125 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:02.746695 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:02.746702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:02.746709 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:02.747129 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:02.747329 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:02.748355 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0929 15:58:02.748629 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":286,"threads_started":1}
I20260929 15:58:02.750226 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:02.727470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:43581 (local address 127.23.222.65:54535)
0929 15:58:02.728468 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:02.728513 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:02.730409 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:02.731329 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:02.731339 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:02.731939 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:02.732592 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:02.732613 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.734174 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.734179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:02.735600 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:02.735616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:02.736909 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:02.736915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:02.737139 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:02.737901 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:02.737926 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:02.740613 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:02.745209 (+ 4596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:02.745218 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:02.745235 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:02.745763 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:02.746278 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:02.746282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:02.746284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:02.746530 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:02.747389 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:02.747397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:02.747886 (+ 489us) client_negotiation.cc:769] Sending connection context
0929 15:58:02.748174 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 15:58:02.748801 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":429,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
I20260929 15:58:02.752409 30254 heartbeater.cc:347] Connected to a master server at 127.23.222.126:43581
I20260929 15:58:02.752839 30254 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:02.753719 30254 heartbeater.cc:511] Master 127.23.222.126:43581 requested a full tablet report, sending...
I20260929 15:58:02.756345 30065 ts_manager.cc:194] Registered new tserver with Master: aa9587efc4ef42a298d968ccb568befb (127.23.222.65:44151)
I20260929 15:58:02.758693 30065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:54535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:58:02.779038 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:43581
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:03.139596 30260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:03.139866 30260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:03.139969 30260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:03.147190 30260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:03.147428 30260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:03.147675 30260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:58:03.158005 30260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:43581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:03.159382 30260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:03.160916 30260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:03.174265 30268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:03.174396 30265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:03.174309 30266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:03.175068 30260 server_base.cc:1034] running on GCE node
I20260929 15:58:03.175794 30260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:03.176795 30260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:03.178054 30260 hybrid_clock.cc:648] HybridClock initialized: now 1790697483177990 us; error 45 us; skew 500 ppm
I20260929 15:58:03.178421 30260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.181788 30260 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:58:03.183357 30260 webserver.cc:460] Webserver started at http://127.23.222.66:41247/ using document root <none> and password file <none>
I20260929 15:58:03.184088 30260 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:03.184243 30260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:03.184569 30260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:03.187690 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "43bf98fcc2b846cca62fc2b3da7a248e"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-cq88"
server_key: "f9ebe3ddfe3d3d0c352dd2813d6fa511"
server_key_iv: "8c3c1c2881e42674aeb69e5d2f587553"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.188519 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "43bf98fcc2b846cca62fc2b3da7a248e"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-cq88"
server_key: "f9ebe3ddfe3d3d0c352dd2813d6fa511"
server_key_iv: "8c3c1c2881e42674aeb69e5d2f587553"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.194139 30260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:58:03.198026 30275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.199795 30260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:58:03.199958 30260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "43bf98fcc2b846cca62fc2b3da7a248e"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-cq88"
server_key: "f9ebe3ddfe3d3d0c352dd2813d6fa511"
server_key_iv: "8c3c1c2881e42674aeb69e5d2f587553"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.200183 30260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:03.217597 30260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:03.222239 30260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:03.222496 30260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:03.223551 30260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:03.225066 30260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:03.225140 30260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.225277 30260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:03.225337 30260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.258349 30260 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:41835
I20260929 15:58:03.258391 30387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:41835 every 8 connection(s)
I20260929 15:58:03.259807 30260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:03.260542 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 30260
I20260929 15:58:03.260713 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:03.261252 24441 external_mini_cluster.cc:1428] Setting key d3c1c9f7d41717261f07f8ab17458f3b
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
I20260929 15:58:03.285097 30390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.264348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:55885 (local address 127.23.222.126:43581)
0929 15:58:03.264805 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.264812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.266087 (+ 1275us) server_negotiation.cc:407] Connection header received
0929 15:58:03.269001 (+ 2914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.269006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.269084 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.269179 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.271514 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.272317 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.273990 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.274448 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.274609 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.278244 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:03.278249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:03.278252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:03.278286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:03.281325 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.282279 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.282284 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.282286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.282434 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.283018 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.283023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.283025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.283400 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:03.283573 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.284401 (+ 828us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.284660 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":219,"threads_started":1}
I20260929 15:58:03.286206 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.265028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:43581 (local address 127.23.222.66:55885)
0929 15:58:03.265903 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:03.265960 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:03.268683 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:03.269473 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:03.269487 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:03.270322 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:03.271036 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.271331 (+ 295us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.272486 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.272492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.273832 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.273845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.274710 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.274717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.274963 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.275730 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:03.275754 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:03.277983 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:03.281527 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:03.281538 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:03.281566 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:03.282119 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:03.282564 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:03.282570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:03.282574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:03.282858 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:03.283544 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:03.283553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:03.283984 (+ 431us) client_negotiation.cc:769] Sending connection context
0929 15:58:03.284278 (+ 294us) client_negotiation.cc:240] Negotiation successful
0929 15:58:03.284886 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":328,"threads_started":1}
I20260929 15:58:03.288003 30388 heartbeater.cc:347] Connected to a master server at 127.23.222.126:43581
I20260929 15:58:03.288403 30388 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:03.289218 30388 heartbeater.cc:511] Master 127.23.222.126:43581 requested a full tablet report, sending...
I20260929 15:58:03.290864 30065 ts_manager.cc:194] Registered new tserver with Master: 43bf98fcc2b846cca62fc2b3da7a248e (127.23.222.66:41835)
I20260929 15:58:03.291777 30065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:55885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:58:03.311123 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:43581
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:03.652894 30394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:03.653198 30394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:03.653358 30394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:03.661414 30394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:03.661540 30394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:03.661718 30394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:58:03.672039 30394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:43581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:03.673487 30394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:03.674930 30394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:03.689384 30399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:03.689425 30402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:03.689464 30400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:03.690353 30394 server_base.cc:1034] running on GCE node
I20260929 15:58:03.691134 30394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:03.692240 30394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:03.693471 30394 hybrid_clock.cc:648] HybridClock initialized: now 1790697483693416 us; error 47 us; skew 500 ppm
I20260929 15:58:03.693845 30394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.697418 30394 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:58:03.699004 30394 webserver.cc:460] Webserver started at http://127.23.222.67:44327/ using document root <none> and password file <none>
I20260929 15:58:03.700057 30394 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:03.700224 30394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:03.700603 30394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:03.704059 30394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "d4ccbb0dc8ad4ce4bd143e9afce27181"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-cq88"
server_key: "d64f3ce9054c783dc1e6a49c15afd796"
server_key_iv: "843b895d94065a3c18946415f70ac331"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.705017 30394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "d4ccbb0dc8ad4ce4bd143e9afce27181"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-cq88"
server_key: "d64f3ce9054c783dc1e6a49c15afd796"
server_key_iv: "843b895d94065a3c18946415f70ac331"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.710587 30394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:58:03.714530 30409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.716202 30394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:03.716351 30394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "d4ccbb0dc8ad4ce4bd143e9afce27181"
format_stamp: "Formatted at 2026-09-29 15:58:03 on dist-test-slave-cq88"
server_key: "d64f3ce9054c783dc1e6a49c15afd796"
server_key_iv: "843b895d94065a3c18946415f70ac331"
server_key_version: "encryptionkey@0"
I20260929 15:58:03.716539 30394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:03.738389 30394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:03.742882 30394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:03.743168 30394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:03.744212 30394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:03.745594 30394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:03.745683 30394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.745784 30394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:03.745831 30394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:03.763090 30254 heartbeater.cc:503] Master 127.23.222.126:43581 was elected leader, sending a full tablet report...
I20260929 15:58:03.781152 30394 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:35555
I20260929 15:58:03.781199 30521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:35555 every 8 connection(s)
I20260929 15:58:03.782508 30394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:03.783046 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 30394
I20260929 15:58:03.783231 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:03.783869 24441 external_mini_cluster.cc:1428] Setting key fc6516c32f665217ebcc8eb63f85fdbc
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:03.803619 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.785603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:58313 (local address 127.23.222.126:43581)
0929 15:58:03.786083 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.786091 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.787194 (+ 1103us) server_negotiation.cc:407] Connection header received
0929 15:58:03.789097 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.789101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.789178 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.789272 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.790857 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.791762 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.793310 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.793712 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.793809 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.797634 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:03.797639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:03.797643 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:03.797678 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:03.800588 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.801380 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.801383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.801385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.801498 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.801922 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.801926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.801928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.802221 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:03.802360 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.803076 (+ 716us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.803321 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":244,"threads_started":1}
I20260929 15:58:03.804759 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.786139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:43581 (local address 127.23.222.67:58313)
0929 15:58:03.787025 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:03.787075 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:03.788819 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:03.789512 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:03.789520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:03.790077 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:03.790670 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.790688 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.791902 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.791906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.793182 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:03.793194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.793917 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.793922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:03.794140 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.794845 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:03.794872 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:03.797336 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:03.800756 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:03.800764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:03.800780 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:03.801263 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:03.801606 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:03.801609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:03.801611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:03.801796 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:03.802342 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:03.802348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:03.802711 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 15:58:03.802956 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 15:58:03.803468 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":382,"threads_started":1}
I20260929 15:58:03.806377 30522 heartbeater.cc:347] Connected to a master server at 127.23.222.126:43581
I20260929 15:58:03.806736 30522 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:03.807587 30522 heartbeater.cc:511] Master 127.23.222.126:43581 requested a full tablet report, sending...
I20260929 15:58:03.809151 30065 ts_manager.cc:194] Registered new tserver with Master: d4ccbb0dc8ad4ce4bd143e9afce27181 (127.23.222.67:35555)
I20260929 15:58:03.809922 30065 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:58313
I20260929 15:58:03.813093 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/test-admin.keytab.
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.842511 24441 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 15:58:03.848228 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:03.845392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.23.222.65:44151)
0929 15:58:03.846091 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.846098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.846126 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:58:03.846190 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.846197 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.846415 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.846561 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.847775 (+ 1214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39172: BlockingRecv error: recv got EOF from 127.0.0.1:39172 (error 108)
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":381,"threads_started":1}
W20260929 15:58:03.848273 24441 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.849980 24441 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.851362 24441 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:58:03.869182 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.853015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.23.222.65:44151)
0929 15:58:03.853222 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.853228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.853447 (+ 219us) server_negotiation.cc:407] Connection header received
0929 15:58:03.853596 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.853600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.853676 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.853761 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.854732 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.855787 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.857195 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.857908 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.858039 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.862955 (+ 4916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:03.862971 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:03.862986 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:03.863037 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:03.866107 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.866929 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.866938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.866940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.867069 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.867509 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.867524 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.867527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.867917 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:03.868258 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.868606 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.868888 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260929 15:58:03.875550 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:03.873797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.23.222.65:44151)
0929 15:58:03.874035 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.874041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.874204 (+ 163us) server_negotiation.cc:407] Connection header received
0929 15:58:03.874347 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.874351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.874425 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.874506 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.875299 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39192: BlockingRecv error: recv got EOF from 127.0.0.1:39192 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260929 15:58:03.875722 24441 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.877656 24441 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.879722 24441 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:58:03.898332 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.883752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39200 (local address 127.23.222.65:44151)
0929 15:58:03.884034 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.884043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.884370 (+ 327us) server_negotiation.cc:407] Connection header received
0929 15:58:03.884598 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.884603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.884683 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.884808 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.886130 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.886951 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.888318 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.888804 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.888948 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.892792 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:03.892798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:03.892801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:03.892839 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:03.895570 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.896335 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.896339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.896342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.896473 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.896945 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.896949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.896951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.897238 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:03.897435 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.897842 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.898073 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
W20260929 15:58:03.903038 30530 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:03.901461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.23.222.65:44151)
0929 15:58:03.901668 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.901674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.901771 (+ 97us) server_negotiation.cc:407] Connection header received
0929 15:58:03.901907 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.901911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.901988 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.902071 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.902781 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39216: BlockingRecv error: recv got EOF from 127.0.0.1:39216 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 15:58:03.903290 24441 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790697431711191-24441-0/krb5kdc/krb5cc' of type FILE
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.904899 24441 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:03.906328 24441 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 15:58:03 dist-test-slave-cq88 krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:58:03.922298 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.907870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.23.222.65:44151)
0929 15:58:03.908158 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.908168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.908517 (+ 349us) server_negotiation.cc:407] Connection header received
0929 15:58:03.908780 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.908784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.908846 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.908987 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.909974 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.910690 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.912022 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.912455 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.912595 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.916547 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:03.916553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:03.916556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:03.916597 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:03.919508 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.920277 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.920288 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.920290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.920448 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.920834 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.920840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.920844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.921138 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:03.921458 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.921731 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.921951 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:58:03.924177 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 30126
I20260929 15:58:03.936983 30530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:03.925060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.23.222.65:44151)
0929 15:58:03.925303 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:03.925311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:03.925478 (+ 167us) server_negotiation.cc:407] Connection header received
0929 15:58:03.925774 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:03.925779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:03.925853 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:03.925968 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:03.926980 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.927831 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.929095 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:03.929586 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:03.929702 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:03.931880 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:03.931885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:03.931889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:03.931932 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:03.934572 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.935211 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.935215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.935218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.935447 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:03.935826 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:03.935830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:03.935832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:03.936110 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:03.936311 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:03.936564 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 15:58:03.936726 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 15:58:03.937644 30249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:04.021780 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 30126
I20260929 15:58:04.038815 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 30260
Sep 29 15:58:04 dist-test-slave-cq88 krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:58:04.055167 30543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.039843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.23.222.66:41835)
0929 15:58:04.040464 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.040472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.040498 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:58:04.040561 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.040568 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.040788 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.040923 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:04.041912 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.042943 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.044397 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.044988 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.045098 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.049161 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:04.049175 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:04.049190 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:04.049226 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:04.052385 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.053158 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.053165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.053168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.053301 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.053746 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.053749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.053751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.054090 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:04.054309 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.054666 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.054871 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":337,"threads_started":1}
I20260929 15:58:04.056604 30383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:04.140424 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 30260
I20260929 15:58:04.156621 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 30394
Sep 29 15:58:04 dist-test-slave-cq88 krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:58:04.172863 30545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:04.157754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.23.222.67:35555)
0929 15:58:04.158422 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:04.158430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:04.158460 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:58:04.158524 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:04.158530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:04.158739 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:04.158871 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:04.160052 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.161154 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.162443 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:04.163012 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:04.163154 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:04.167299 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:04.167314 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:04.167331 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:04.167371 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:04.170044 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.170764 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.170772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.170774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.170896 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:04.171302 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:04.171309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:04.171313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:04.171671 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:04.172013 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:04.172282 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 15:58:04.172544 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":378,"threads_started":1}
I20260929 15:58:04.174410 30517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:04.264534 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 30394
I20260929 15:58:04.281633 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 30033
I20260929 15:58:04.282894 30095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:04.388150 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 30033
2026-09-29T15:58:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4058 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:04 dist-test-slave-cq88 krb5kdc[30550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:04 dist-test-slave-cq88 krb5kdc[30550](info): set up 2 sockets
Sep 29 15:58:04 dist-test-slave-cq88 krb5kdc[30550](info): commencing operation
krb5kdc: starting...
W20260929 15:58:05.499145 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:58:05 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.126.keytab.
Entry for principal oryx/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.126.keytab.
I20260929 15:58:05.727820 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42641
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.126.keytab
--principal=oryx/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:42641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:06.062942 30563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:06.063364 30563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:06.063478 30563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:06.070842 30563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:06.070961 30563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:06.071008 30563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:06.071061 30563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:06.071106 30563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:06.081311 30563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:42641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.126.keytab
--principal=oryx/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:06.082641 30563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:06.083937 30563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:06.096279 30569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:06.096302 30571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:06.096616 30568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:06.097123 30563 server_base.cc:1034] running on GCE node
I20260929 15:58:06.097909 30563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:06.099427 30563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:06.100677 30563 hybrid_clock.cc:648] HybridClock initialized: now 1790697486100603 us; error 64 us; skew 500 ppm
I20260929 15:58:06.101037 30563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:06 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:06.104336 30563 init.cc:377] Logged in from keytab as oryx/127.23.222.126@KRBTEST.COM (short username oryx)
I20260929 15:58:06.105863 30563 webserver.cc:460] Webserver started at http://127.23.222.126:35411/ using document root <none> and password file <none>
I20260929 15:58:06.106555 30563 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:06.106659 30563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:06.106900 30563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:06.109915 30563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-cq88"
server_key: "3c34ad39b33beff6c12245d8fcb09b82"
server_key_iv: "0ccc9f78899732fc95272742ffbaa859"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.110657 30563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-cq88"
server_key: "3c34ad39b33beff6c12245d8fcb09b82"
server_key_iv: "0ccc9f78899732fc95272742ffbaa859"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.116185 30563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 15:58:06.119870 30578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.121330 30563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:58:06.121495 30563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-cq88"
server_key: "3c34ad39b33beff6c12245d8fcb09b82"
server_key_iv: "0ccc9f78899732fc95272742ffbaa859"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.121675 30563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:06.137422 30563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:06.142103 30563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:06.142354 30563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:06.162557 30563 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:42641
I20260929 15:58:06.162616 30629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:42641 every 8 connection(s)
I20260929 15:58:06.164233 30563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:06.168226 30630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:06.169682 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 30563
I20260929 15:58:06.169939 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:58:06.170694 24441 external_mini_cluster.cc:1428] Setting key 161e87139911c5dceb086ff2d69ab1a8
I20260929 15:58:06.184988 30630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: Bootstrap starting.
Sep 29 15:58:06 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:06.189323 30630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:06.190423 30630 log.cc:824] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:06.193631 30630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: No bootstrap required, opened a new log
I20260929 15:58:06.197894 30633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:06.173000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.23.222.126:42641)
0929 15:58:06.173987 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:06.174003 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:06.174070 (+ 67us) server_negotiation.cc:407] Connection header received
0929 15:58:06.175921 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:06.175957 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:06.176545 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:06.177000 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:06.178933 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:06.181452 (+ 2519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:06.183577 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:06.184776 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:06.184990 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:06.189978 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:06.189988 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:06.190005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:06.190047 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:06.193389 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:06.194225 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:06.194233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:06.194239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:06.194419 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:06.194840 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:06.194845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:06.194848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:06.195510 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:06.195869 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:06.196338 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 15:58:06.196802 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":471,"threads_started":1}
I20260929 15:58:06.202381 30630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } }
I20260929 15:58:06.202781 30630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:06.202904 30630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d82bc0a64f455b9e4c9bacf9b78ae0, State: Initialized, Role: FOLLOWER
I20260929 15:58:06.203555 30630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } }
I20260929 15:58:06.203814 30630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:06.203969 30630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:06.204151 30630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:06.206444 30630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } }
I20260929 15:58:06.206971 30630 leader_election.cc:302] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d82bc0a64f455b9e4c9bacf9b78ae0; no voters:
I20260929 15:58:06.207686 30630 leader_election.cc:288] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:06.207832 30635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:06.208717 30635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 1 LEADER]: Becoming Leader. State: Replica: 78d82bc0a64f455b9e4c9bacf9b78ae0, State: Running, Role: LEADER
I20260929 15:58:06.209394 30635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } }
I20260929 15:58:06.210014 30630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:06.212270 30636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } }
I20260929 15:58:06.212635 30636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:06.212926 30637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d82bc0a64f455b9e4c9bacf9b78ae0. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } }
I20260929 15:58:06.213526 30637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:06.214983 30640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:06.220475 30640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:06.230602 30640 catalog_manager.cc:1347] Generated new cluster ID: 0bb7066fd387453e8024b8fb435387d6
I20260929 15:58:06.230732 30640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:06.241693 30640 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:06.243760 30640 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:06.251890 30640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: Generated new TSK 0
I20260929 15:58:06.252765 30640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab.
Entry for principal oryx/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab.
I20260929 15:58:06.303746 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42641
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab
--principal=oryx/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:06.642642 30656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:06.642951 30656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:06.643117 30656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:06.650733 30656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:06.650863 30656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:06.651049 30656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:58:06.660909 30656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab
--principal=oryx/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:06.662302 30656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:06.663713 30656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:06.677081 30661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:06.677175 30664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:06.677122 30662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:06.678062 30656 server_base.cc:1034] running on GCE node
I20260929 15:58:06.678774 30656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:06.679930 30656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:06.681171 30656 hybrid_clock.cc:648] HybridClock initialized: now 1790697486681125 us; error 45 us; skew 500 ppm
I20260929 15:58:06.681529 30656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:06 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:06.684856 30656 init.cc:377] Logged in from keytab as oryx/127.23.222.65@KRBTEST.COM (short username oryx)
I20260929 15:58:06.686384 30656 webserver.cc:460] Webserver started at http://127.23.222.65:39833/ using document root <none> and password file <none>
I20260929 15:58:06.687085 30656 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:06.687191 30656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:06.687558 30656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:06.690528 30656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "83aec33916c44295876e27cc9e18e527"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-cq88"
server_key: "b162d95edf22a67eb96afea39564d886"
server_key_iv: "8904114750686e2b92d204c803ee28cf"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.691825 30656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "83aec33916c44295876e27cc9e18e527"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-cq88"
server_key: "b162d95edf22a67eb96afea39564d886"
server_key_iv: "8904114750686e2b92d204c803ee28cf"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.697722 30656 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:58:06.701552 30671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.703079 30656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:06.703320 30656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "83aec33916c44295876e27cc9e18e527"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-cq88"
server_key: "b162d95edf22a67eb96afea39564d886"
server_key_iv: "8904114750686e2b92d204c803ee28cf"
server_key_version: "encryptionkey@0"
I20260929 15:58:06.703523 30656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:06.734967 30656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:06.739671 30656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:06.739917 30656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:06.740975 30656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:06.742700 30656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:06.742779 30656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.742906 30656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:06.742991 30656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:06.778329 30656 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:33817
I20260929 15:58:06.778376 30783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:33817 every 8 connection(s)
I20260929 15:58:06.779825 30656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:06.789113 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 30656
I20260929 15:58:06.789318 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:06.789991 24441 external_mini_cluster.cc:1428] Setting key 9b48f374f5088c549340d489bf4ef2ac
Sep 29 15:58:06 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:06.803023 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:06.782716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:45795 (local address 127.23.222.126:42641)
0929 15:58:06.783200 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:06.783205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:06.784215 (+ 1010us) server_negotiation.cc:407] Connection header received
0929 15:58:06.786102 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:06.786107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:06.786203 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:06.786327 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:06.787923 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:06.788716 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:06.790764 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:06.791292 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:06.791434 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:06.795989 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:06.795999 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:06.796007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:06.796059 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:06.799195 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:06.800191 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:06.800200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:06.800207 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:06.800369 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:06.800889 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:06.800897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:06.800902 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:06.801304 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:06.801461 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:06.802266 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 15:58:06.802530 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":269,"threads_started":1}
I20260929 15:58:06.804154 30787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:06.783193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.65:45795)
0929 15:58:06.784013 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:06.784081 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:06.785796 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:06.786582 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:06.786591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:06.787130 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:06.787730 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:06.787747 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:06.788946 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:06.788950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:06.790570 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:06.790589 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:06.791714 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:06.791720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:06.792112 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:06.793108 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:06.793136 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:06.795663 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:06.799477 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:06.799486 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:06.799524 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:06.800027 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:06.800517 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:06.800521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:06.800524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:06.800753 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:06.801445 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:06.801452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:06.801813 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 15:58:06.802096 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 15:58:06.802549 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":293,"threads_started":1}
I20260929 15:58:06.806212 30784 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42641
I20260929 15:58:06.806581 30784 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:06.807694 30784 heartbeater.cc:511] Master 127.23.222.126:42641 requested a full tablet report, sending...
I20260929 15:58:06.810659 30595 ts_manager.cc:194] Registered new tserver with Master: 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817)
WARNING: no policy specified for oryx/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 15:58:06.812340 30595 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:45795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab.
Entry for principal oryx/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab.
I20260929 15:58:06.842816 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42641
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab
--principal=oryx/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:07.175320 30790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:07.175631 30790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:07.175797 30790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:07.183367 30790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:07.183511 30790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:07.183698 30790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:58:07.193764 30790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab
--principal=oryx/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:07.195171 30790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:07.196579 30790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:07.210609 30798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:07.210614 30796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:07.210547 30795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:07.212699 30790 server_base.cc:1034] running on GCE node
I20260929 15:58:07.213362 30790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:07.214389 30790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:07.215590 30790 hybrid_clock.cc:648] HybridClock initialized: now 1790697487215521 us; error 57 us; skew 500 ppm
I20260929 15:58:07.215948 30790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:07 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697487, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:07.219709 30790 init.cc:377] Logged in from keytab as oryx/127.23.222.66@KRBTEST.COM (short username oryx)
I20260929 15:58:07.221333 30790 webserver.cc:460] Webserver started at http://127.23.222.66:44421/ using document root <none> and password file <none>
I20260929 15:58:07.222081 30790 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:07.222208 30790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:07.222560 30790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:07.225836 30790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "f3f02f25c6ca4061902c1fb1a305d7c6"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-cq88"
server_key: "d543414bfdb050714b6169048aff1821"
server_key_iv: "c23e2b11f83a109202e2b6693d7bf00c"
server_key_version: "encryptionkey@0"
I20260929 15:58:07.226713 30790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "f3f02f25c6ca4061902c1fb1a305d7c6"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-cq88"
server_key: "d543414bfdb050714b6169048aff1821"
server_key_iv: "c23e2b11f83a109202e2b6693d7bf00c"
server_key_version: "encryptionkey@0"
I20260929 15:58:07.232218 30790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 15:58:07.235954 30805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:07.237352 30790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:07.237543 30790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "f3f02f25c6ca4061902c1fb1a305d7c6"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-cq88"
server_key: "d543414bfdb050714b6169048aff1821"
server_key_iv: "c23e2b11f83a109202e2b6693d7bf00c"
server_key_version: "encryptionkey@0"
I20260929 15:58:07.237756 30790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:07.252637 30790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:07.256907 30790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:07.257217 30790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:07.258272 30790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:07.259719 30790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:07.259819 30790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:07.259970 30790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:07.260046 30790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:07.294287 30790 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:38077
I20260929 15:58:07.294335 30917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:38077 every 8 connection(s)
I20260929 15:58:07.295698 30790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:07.304484 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 30790
I20260929 15:58:07.304680 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:07.305238 24441 external_mini_cluster.cc:1428] Setting key ff696b61d79a7a5b614b432ea0d5320b
Sep 29 15:58:07 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697487, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:07.318976 30786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.298400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:51553 (local address 127.23.222.126:42641)
0929 15:58:07.298603 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.298609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.300028 (+ 1419us) server_negotiation.cc:407] Connection header received
0929 15:58:07.301974 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.301978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.302059 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.302153 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.303842 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:07.304896 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:07.306886 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:07.307594 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:07.307697 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:07.312240 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.312250 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.312258 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:07.312305 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.315478 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:07.316502 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:07.316506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:07.316508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:07.316646 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:07.317199 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:07.317204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:07.317206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:07.317495 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.317649 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.318571 (+ 922us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.318738 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:58:07.320580 30920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.298972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.66:51553)
0929 15:58:07.299839 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.299891 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:07.301676 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:07.302420 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:07.302428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:07.302969 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:07.303655 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:07.303672 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:07.305134 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:07.305138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:07.306733 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:07.306754 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:07.307816 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:07.307824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:07.308136 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:07.308926 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:07.308963 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:07.311976 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:07.315637 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:07.315650 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:07.315671 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:07.316314 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:07.316772 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:07.316780 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:07.316786 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:07.317062 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:07.317656 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:07.317667 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:07.318175 (+ 508us) client_negotiation.cc:769] Sending connection context
0929 15:58:07.318477 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 15:58:07.319061 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":386,"threads_started":1}
I20260929 15:58:07.322018 30918 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42641
I20260929 15:58:07.322336 30918 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:07.323096 30918 heartbeater.cc:511] Master 127.23.222.126:42641 requested a full tablet report, sending...
I20260929 15:58:07.324692 30595 ts_manager.cc:194] Registered new tserver with Master: f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077)
I20260929 15:58:07.325562 30595 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:51553
WARNING: no policy specified for oryx/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab.
Entry for principal oryx/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab.
I20260929 15:58:07.358517 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42641
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab
--principal=oryx/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:07.688503 30923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:07.688858 30923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:07.689042 30923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:07.696653 30923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:07.696847 30923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:07.697045 30923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:58:07.707490 30923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab
--principal=oryx/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:07.708984 30923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:07.710547 30923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:07.726107 30928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:07.726104 30929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:07.726356 30931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:07.726915 30923 server_base.cc:1034] running on GCE node
I20260929 15:58:07.727731 30923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:07.728708 30923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:07.729929 30923 hybrid_clock.cc:648] HybridClock initialized: now 1790697487729875 us; error 45 us; skew 500 ppm
I20260929 15:58:07.730290 30923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:07 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697487, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:07.741001 30923 init.cc:377] Logged in from keytab as oryx/127.23.222.67@KRBTEST.COM (short username oryx)
I20260929 15:58:07.742797 30923 webserver.cc:460] Webserver started at http://127.23.222.67:33067/ using document root <none> and password file <none>
I20260929 15:58:07.743618 30923 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:07.743770 30923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:07.744095 30923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:07.747090 30923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "495c68ea69d948d2888be1132f244819"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-cq88"
server_key: "c8fb6b951f96265f1fde032523a38f98"
server_key_iv: "1c099cb0d6010d6632b60ef5f2f3f7be"
server_key_version: "encryptionkey@0"
I20260929 15:58:07.748019 30923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "495c68ea69d948d2888be1132f244819"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-cq88"
server_key: "c8fb6b951f96265f1fde032523a38f98"
server_key_iv: "1c099cb0d6010d6632b60ef5f2f3f7be"
server_key_version: "encryptionkey@0"
I20260929 15:58:07.753618 30923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:07.757618 30938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:07.759359 30923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:07.759567 30923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "495c68ea69d948d2888be1132f244819"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-cq88"
server_key: "c8fb6b951f96265f1fde032523a38f98"
server_key_iv: "1c099cb0d6010d6632b60ef5f2f3f7be"
server_key_version: "encryptionkey@0"
I20260929 15:58:07.759804 30923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:07.777319 30923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:07.782485 30923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:07.782824 30923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:07.784080 30923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:07.785565 30923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:07.785679 30923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:07.785843 30923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:07.785916 30923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:07.815588 30784 heartbeater.cc:503] Master 127.23.222.126:42641 was elected leader, sending a full tablet report...
I20260929 15:58:07.821751 30923 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:38233
I20260929 15:58:07.821717 31050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:38233 every 8 connection(s)
I20260929 15:58:07.823041 30923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:07.830360 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 30923
I20260929 15:58:07.830525 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:07.831070 24441 external_mini_cluster.cc:1428] Setting key e2d141bf35bc0c7535f4290f0989a5b2
Sep 29 15:58:07 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697487, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:07.843360 31053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.825792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:48977 (local address 127.23.222.126:42641)
0929 15:58:07.826253 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:07.826257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:07.827265 (+ 1008us) server_negotiation.cc:407] Connection header received
0929 15:58:07.829070 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:07.829075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:07.829168 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:07.829287 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:07.830960 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:07.831876 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:07.833703 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:07.834289 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:07.834426 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:07.837892 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:07.837897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:07.837900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:07.837930 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:07.840427 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:07.841169 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:07.841173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:07.841175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:07.841304 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:07.841693 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:07.841696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:07.841698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:07.841969 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:07.842159 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:07.842837 (+ 678us) server_negotiation.cc:299] Negotiation successful
0929 15:58:07.843024 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":254,"threads_started":1}
I20260929 15:58:07.844331 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:07.826317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.67:48977)
0929 15:58:07.827080 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:07.827124 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:07.828770 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:07.829537 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:07.829546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:07.830081 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:07.830784 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:07.830806 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:07.832040 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:07.832044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:07.833546 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:07.833575 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:07.834541 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:07.834549 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:07.834799 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:07.835515 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:07.835543 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:07.837706 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:07.840561 (+ 2855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:07.840569 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:07.840582 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:07.841020 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:07.841402 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:07.841405 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:07.841408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:07.841599 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:07.842104 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:07.842110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:07.842481 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 15:58:07.842737 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 15:58:07.843130 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":289,"threads_started":1}
I20260929 15:58:07.845826 31051 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42641
I20260929 15:58:07.846163 31051 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:07.847021 31051 heartbeater.cc:511] Master 127.23.222.126:42641 requested a full tablet report, sending...
I20260929 15:58:07.848539 30594 ts_manager.cc:194] Registered new tserver with Master: 495c68ea69d948d2888be1132f244819 (127.23.222.67:38233)
I20260929 15:58:07.849452 30594 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:48977
I20260929 15:58:07.859462 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.222.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.125.keytab.
Entry for principal oryx/127.23.222.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.125.keytab.
I20260929 15:58:07.911602 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.125:43047
--webserver_interface=127.23.222.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.125.keytab
--principal=oryx/127.23.222.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:42641,127.23.222.125:43047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:08.249701 31057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:08.250080 31057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:08.250247 31057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:08.257548 31057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:08.257718 31057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:08.257833 31057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:08.257939 31057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:08.258028 31057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:08.268383 31057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:42641,127.23.222.125:43047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.125.keytab
--principal=oryx/127.23.222.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.125:43047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.222.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
W20260929 15:58:08.268831 31057 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.222.126:42641,127.23.222.125:43047'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 15:58:08.269934 31057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:08.271328 31057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:08.283617 31062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:08.283581 31065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:08.283999 31057 server_base.cc:1034] running on GCE node
W20260929 15:58:08.283547 31063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:08.285095 31057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:08.286484 31057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:08.287741 31057 hybrid_clock.cc:648] HybridClock initialized: now 1790697488287677 us; error 52 us; skew 500 ppm
I20260929 15:58:08.288111 31057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:08 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697488, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:08.291533 31057 init.cc:377] Logged in from keytab as oryx/127.23.222.125@KRBTEST.COM (short username oryx)
I20260929 15:58:08.293046 31057 webserver.cc:460] Webserver started at http://127.23.222.125:42201/ using document root <none> and password file <none>
I20260929 15:58:08.293897 31057 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:08.293989 31057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:08.294301 31057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:08.297448 31057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data/instance:
uuid: "ba8d5e94f773430f962bb7b2dfc4633f"
format_stamp: "Formatted at 2026-09-29 15:58:08 on dist-test-slave-cq88"
server_key: "e9661091c7e13f3115848670890b852a"
server_key_iv: "8b0c38d1dd75c2dad0afd6f11f8f52fb"
server_key_version: "encryptionkey@0"
I20260929 15:58:08.298358 31057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal/instance:
uuid: "ba8d5e94f773430f962bb7b2dfc4633f"
format_stamp: "Formatted at 2026-09-29 15:58:08 on dist-test-slave-cq88"
server_key: "e9661091c7e13f3115848670890b852a"
server_key_iv: "8b0c38d1dd75c2dad0afd6f11f8f52fb"
server_key_version: "encryptionkey@0"
I20260929 15:58:08.303813 31057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:58:08.307895 31072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:08.309506 31057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:58:08.309695 31057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal
uuid: "ba8d5e94f773430f962bb7b2dfc4633f"
format_stamp: "Formatted at 2026-09-29 15:58:08 on dist-test-slave-cq88"
server_key: "e9661091c7e13f3115848670890b852a"
server_key_iv: "8b0c38d1dd75c2dad0afd6f11f8f52fb"
server_key_version: "encryptionkey@0"
I20260929 15:58:08.309894 31057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:08.328241 30918 heartbeater.cc:503] Master 127.23.222.126:42641 was elected leader, sending a full tablet report...
I20260929 15:58:08.352877 31057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:08.357496 31057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:08.357795 31057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:08.377964 31057 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.125:43047
I20260929 15:58:08.377991 31123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.125:43047 every 8 connection(s)
I20260929 15:58:08.379492 31057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
I20260929 15:58:08.383167 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 31057
I20260929 15:58:08.383571 31124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:08.393568 31124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:08 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.125@KRBTEST.COM
Sep 29 15:58:08 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697488, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.125@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:08.408870 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.385683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.23.222.125:43047)
0929 15:58:08.386573 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:08.386586 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:08.386639 (+ 53us) server_negotiation.cc:407] Connection header received
0929 15:58:08.388534 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:08.388559 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:08.388992 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:08.389380 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:08.391317 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.392709 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.394500 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.395329 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.395474 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.400702 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:08.400713 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:08.400727 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:08.400773 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:08.404166 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.405034 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.405040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.405046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.405252 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.405643 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.405648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.405652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.406361 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:08.406658 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:08.407157 (+ 499us) server_negotiation.cc:299] Negotiation successful
0929 15:58:08.407619 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":414,"threads_started":1}
I20260929 15:58:08.412992 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.396675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33622 (local address 127.23.222.126:42641)
0929 15:58:08.398028 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:08.398036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:08.398057 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:58:08.398121 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:08.398129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:08.398216 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:08.398338 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:08.399656 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.400459 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.402127 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.402828 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.402968 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.406795 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:08.406802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:08.406807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:08.406872 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:08.409998 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.410744 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.410749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.410752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.410930 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.411348 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.411352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.411355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.411755 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:08.411927 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:08.412387 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 15:58:08.412610 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":1117,"threads_started":1}
I20260929 15:58:08.413043 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.396565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:33622)
0929 15:58:08.397352 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:08.397412 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:08.397563 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:08.398435 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:08.398443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:08.399150 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:08.399503 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.399516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.400627 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.400630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.401964 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.401981 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.403059 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.403064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.403190 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.403879 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:08.403897 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:08.406548 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:08.410155 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.410164 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.410171 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.410617 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.411048 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.411051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.411054 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.411186 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.411896 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:08.411904 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:08.412074 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 15:58:08.412276 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 15:58:08.412610 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":334,"threads_started":1}
I20260929 15:58:08.414769 31124 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:08 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697488, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.125@KRBTEST.COM for oryx/127.23.222.125@KRBTEST.COM
I20260929 15:58:08.427606 31128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.415501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.0.0.1:39238)
0929 15:58:08.415655 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:08.415670 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:08.415748 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:08.416065 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:08.416068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:08.416352 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:08.416478 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.416485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.417569 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.417573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.418537 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.418549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.419378 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.419383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.419531 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.420114 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:08.420132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:08.422227 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:08.425203 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.425207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.425210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.425685 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.426081 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.426084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.426086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.426207 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.426803 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:08.426806 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:08.426937 (+ 131us) client_negotiation.cc:769] Sending connection context
0929 15:58:08.427056 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 15:58:08.427349 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 15:58:08.427866 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.415653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39238 (local address 127.23.222.125:43047)
0929 15:58:08.415801 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:08.415805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:08.415819 (+ 14us) server_negotiation.cc:407] Connection header received
0929 15:58:08.415855 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:08.415858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:08.415935 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:08.416024 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:08.416682 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.417393 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.418689 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.419129 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.419229 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.422404 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:08.422408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:08.422411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:08.422442 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:08.425068 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.425836 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.425841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.425843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.425971 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.426360 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.426364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.426366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.426656 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:08.426772 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:08.427137 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 15:58:08.427498 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":57}
I20260929 15:58:08.432723 31124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Bootstrap starting.
I20260929 15:58:08.436061 31124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:08.437129 31124 log.cc:824] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:08.440112 31124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: No bootstrap required, opened a new log
I20260929 15:58:08.447460 31124 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } }
I20260929 15:58:08.447839 31124 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:08.447933 31124 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8d5e94f773430f962bb7b2dfc4633f, State: Initialized, Role: FOLLOWER
I20260929 15:58:08.448714 31124 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } }
I20260929 15:58:08.449584 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } } }
I20260929 15:58:08.449941 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:08.450640 31124 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 15:58:08.459584 31142 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:58:08.459741 31142 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:58:08.462431 31057 master_runner.cc:428] Detected that this master ba8d5e94f773430f962bb7b2dfc4633f is joining an existing cluster
I20260929 15:58:08.462539 31057 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.222.125:43047
I20260929 15:58:08.485625 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.468009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39250 (local address 127.23.222.125:43047)
0929 15:58:08.468191 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:08.468196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:08.469826 (+ 1630us) server_negotiation.cc:407] Connection header received
0929 15:58:08.470005 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:08.470013 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:08.470090 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:08.470184 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:08.471438 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.472542 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.474450 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.475283 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.475401 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.478131 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:08.478140 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:08.478148 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:08.478194 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:08.482212 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.483027 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.483039 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.483044 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.483231 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.483677 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.483685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.483691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.484094 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:08.484253 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:08.484823 (+ 570us) server_negotiation.cc:299] Negotiation successful
0929 15:58:08.485078 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":55}
I20260929 15:58:08.486236 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.466668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.23.222.126:42641)
0929 15:58:08.466930 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:08.466940 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:08.467538 (+ 598us) server_negotiation.cc:407] Connection header received
0929 15:58:08.467804 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:08.467812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:08.467913 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:08.467992 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:08.470571 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.471731 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.473865 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.474582 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.474740 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.478981 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:08.478989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:08.478994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:08.479042 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:08.482657 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.483688 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.483695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.483699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.483877 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.484405 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.484411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.484414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.484796 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:08.484947 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:08.485582 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 15:58:08.485846 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:58:08.487082 31149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.467618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.0.0.1:39250)
0929 15:58:08.469733 (+ 2115us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:08.469748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:08.469887 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:08.470357 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:08.470362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:08.470957 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:08.471293 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.471304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.472738 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.472743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.474298 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.474315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.475735 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.475743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.475998 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.476589 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:08.476618 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:08.477943 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:08.482363 (+ 4420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.482370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.482377 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.482907 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.483397 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.483404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.483410 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.483574 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.484344 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:08.484352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:08.484537 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 15:58:08.486259 (+ 1722us) client_negotiation.cc:240] Negotiation successful
0929 15:58:08.486600 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2023,"thread_start_us":1837,"threads_started":1}
I20260929 15:58:08.487424 31148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.466561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:33632)
0929 15:58:08.467438 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:08.467458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:08.467645 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:08.468209 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:08.468216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:08.469354 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:08.470395 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.470406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.471912 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.471916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.473702 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.473715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.475416 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.475425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.475682 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.476414 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:08.476441 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:08.478728 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:08.482846 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.482851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.482855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.483513 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.484052 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.484057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.484061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.484243 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.485040 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:08.485045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:08.485253 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 15:58:08.486327 (+ 1074us) client_negotiation.cc:240] Negotiation successful
0929 15:58:08.486885 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":168,"spinlock_wait_cycles":31616,"thread_start_us":560,"threads_started":1}
WARNING: no policy specified for oryx/127.23.222.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.124@KRBTEST.COM" created.
Sep 29 15:58:08 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.126@KRBTEST.COM for oryx/127.23.222.125@KRBTEST.COM
I20260929 15:58:08.519367 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.494578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.23.222.125:43047)
0929 15:58:08.494759 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:08.494768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:08.499888 (+ 5120us) server_negotiation.cc:407] Connection header received
0929 15:58:08.500221 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:08.500228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:08.500363 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:08.500499 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:08.502104 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.503285 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.505401 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.506125 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.506265 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.511189 (+ 4924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:08.511203 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:08.511212 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:08.511329 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:08.515678 (+ 4349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.516733 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.516739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.516743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.516920 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.517407 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.517429 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.517435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.517934 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:08.518100 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:08.518763 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 15:58:08.518943 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 15:58:08.520036 31151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.494511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.0.0.1:39264)
0929 15:58:08.499705 (+ 5194us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:08.499751 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:08.499983 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:08.500576 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:08.500586 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:08.501556 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:08.501907 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.501923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.503442 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.503450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.505234 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.505252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.506320 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.506329 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.506578 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.507530 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:08.507559 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:08.510909 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:08.515854 (+ 4945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.515870 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.515881 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.516562 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.517051 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.517057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.517061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.517245 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.518113 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:08.518124 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:08.518420 (+ 296us) client_negotiation.cc:769] Sending connection context
0929 15:58:08.518701 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 15:58:08.519094 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4924,"thread_start_us":269,"threads_started":1}
I20260929 15:58:08.522329 30594 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.222.125:43047: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f"
member_type: NON_VOTER
last_known_addr {
host: "127.23.222.125"
port: 43047
}
attrs {
promote: true
}
}
dest_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0"
cas_config_opid_index: -1
I20260929 15:58:08.523396 30594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.124.keytab.
Entry for principal oryx/127.23.222.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.124.keytab.
I20260929 15:58:08.527261 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.124:41413
--webserver_interface=127.23.222.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.124.keytab
--principal=oryx/127.23.222.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:42641,127.23.222.125:43047,127.23.222.124:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:08.546065 31099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:08.550516 31099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Refusing update from remote peer 78d82bc0a64f455b9e4c9bacf9b78ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 15:58:08.560938 31155 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ba8d5e94f773430f962bb7b2dfc4633f (127.23.222.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:08.562351 31157 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:08.567836 31155 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.222.125:43047
I20260929 15:58:08.568399 31154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:08.568882 31154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:08.570716 31057 master.cc:537] Master@127.23.222.125:43047 shutting down...
I20260929 15:58:08.594210 31099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ba8d5e94f773430f962bb7b2dfc4633f (127.23.222.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:08.599061 31164 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:08.599663 31164 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:08.600106 31161 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:08.600493 31161 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:08.600955 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d82bc0a64f455b9e4c9bacf9b78ae0. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:08.601193 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:08.625837 31057 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 LEARNER]: Raft consensus shutting down.
I20260929 15:58:08.626427 31057 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 LEARNER]: Raft consensus is shut down!
I20260929 15:58:08.626619 31057 tablet_replica.cc:330] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: stopping tablet replica
I20260929 15:58:08.635624 31057 master.cc:552] Master@127.23.222.125:43047 shutdown complete.
I20260929 15:58:08.635807 31057 master_runner.cc:305] Clearing existing system tablet
I20260929 15:58:08.636415 31057 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:08.636595 31057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:08.639932 31057 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 15:58:08.642131 31166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:08.642656 31057 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:08.643030 31057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 15:58:08.643151 31057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal
uuid: "ba8d5e94f773430f962bb7b2dfc4633f"
format_stamp: "Formatted at 2026-09-29 15:58:08 on dist-test-slave-cq88"
server_key: "e9661091c7e13f3115848670890b852a"
server_key_iv: "8b0c38d1dd75c2dad0afd6f11f8f52fb"
server_key_version: "encryptionkey@0"
I20260929 15:58:08.647927 31057 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:08.655926 31057 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 15:58:08.656191 31057 log.cc:1196] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 15:58:08.656738 31057 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Deleting consensus metadata
I20260929 15:58:08.657114 31057 master_runner.cc:315] Copying system tablet from 127.23.222.126:42641
I20260929 15:58:08.663818 31057 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: tablet copy: Beginning tablet copy session from remote peer at address 127.23.222.126:42641
I20260929 15:58:08.683344 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.664880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.23.222.126:42641)
0929 15:58:08.665076 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:08.665083 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:08.667938 (+ 2855us) server_negotiation.cc:407] Connection header received
0929 15:58:08.668239 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:08.668246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:08.668348 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:08.668477 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:08.669538 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.670702 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.672908 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.673698 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.673852 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.676214 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:08.676222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:08.676228 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:08.676275 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:08.680073 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.680910 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.680916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.680920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.681101 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:08.681568 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:08.681575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:08.681578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:08.681969 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:08.682132 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:08.682606 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 15:58:08.682849 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:58:08.683398 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:08.664746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:33648)
0929 15:58:08.667762 (+ 3016us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:08.667793 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:08.668042 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:08.668525 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:08.668530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:08.669055 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:08.669365 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.669377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.670869 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.670874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.672561 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:08.672580 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:08.673896 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:08.673904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:08.674105 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:08.674858 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:08.674885 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:08.675991 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:08.680248 (+ 4257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.680254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.680258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.680761 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.681229 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:08.681234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:08.681238 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:08.681432 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:08.682105 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:08.682109 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:08.682290 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 15:58:08.682437 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 15:58:08.682794 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2835,"thread_start_us":2685,"threads_started":1}
I20260929 15:58:08.684739 30615 tablet_copy_service.cc:140] P 78d82bc0a64f455b9e4c9bacf9b78ae0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ba8d5e94f773430f962bb7b2dfc4633f ({username='oryx', principal='oryx/127.23.222.125@KRBTEST.COM'} at 127.0.0.1:33648)
I20260929 15:58:08.685086 30615 tablet_copy_service.cc:161] P 78d82bc0a64f455b9e4c9bacf9b78ae0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ba8d5e94f773430f962bb7b2dfc4633f at {username='oryx', principal='oryx/127.23.222.125@KRBTEST.COM'} at 127.0.0.1:33648: session id = ba8d5e94f773430f962bb7b2dfc4633f-00000000000000000000000000000000
I20260929 15:58:08.689824 30615 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 15:58:08.692386 31057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 15:58:08.699586 31057 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: tablet copy: Starting download of 0 data blocks...
I20260929 15:58:08.699944 31057 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: tablet copy: Starting download of 1 WAL segments...
I20260929 15:58:08.703015 31057 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 15:58:08.708235 30615 tablet_copy_service.cc:342] P 78d82bc0a64f455b9e4c9bacf9b78ae0: Request end of tablet copy session ba8d5e94f773430f962bb7b2dfc4633f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.222.125@KRBTEST.COM'} at 127.0.0.1:33648
I20260929 15:58:08.708484 30615 tablet_copy_service.cc:433] P 78d82bc0a64f455b9e4c9bacf9b78ae0: ending tablet copy session ba8d5e94f773430f962bb7b2dfc4633f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ba8d5e94f773430f962bb7b2dfc4633f
I20260929 15:58:08.722199 31057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:08.725466 31173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:08.726540 31174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:08.731454 31177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:08.732110 31057 server_base.cc:1034] running on GCE node
I20260929 15:58:08.732582 31057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:08.733376 31057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:08.734782 31057 hybrid_clock.cc:648] HybridClock initialized: now 1790697488734699 us; error 51 us; skew 500 ppm
I20260929 15:58:08.735023 31057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:08 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697488, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:08.737320 31057 init.cc:377] Logged in from keytab as oryx/127.23.222.125@KRBTEST.COM (short username oryx)
I20260929 15:58:08.738691 31057 webserver.cc:460] Webserver started at http://127.23.222.125:41763/ using document root <none> and password file <none>
I20260929 15:58:08.739097 31057 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:08.739200 31057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:08.742731 31057 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 15:58:08.745014 31183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:08.745497 31057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 15:58:08.745641 31057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal
uuid: "ba8d5e94f773430f962bb7b2dfc4633f"
format_stamp: "Formatted at 2026-09-29 15:58:08 on dist-test-slave-cq88"
server_key: "e9661091c7e13f3115848670890b852a"
server_key_iv: "8b0c38d1dd75c2dad0afd6f11f8f52fb"
server_key_version: "encryptionkey@0"
I20260929 15:58:08.745828 31057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:08.767729 31057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:08.772359 31057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:08.806552 31057 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.125:43047
I20260929 15:58:08.806582 31234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.125:43047 every 8 connection(s)
I20260929 15:58:08.807811 31057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
I20260929 15:58:08.816332 31235 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:58:08.822880 31235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Bootstrap starting.
I20260929 15:58:08.839407 31235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:58:08.840225 31235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Bootstrap complete.
I20260929 15:58:08.841416 31235 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } }
I20260929 15:58:08.841722 31235 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ba8d5e94f773430f962bb7b2dfc4633f, State: Initialized, Role: LEARNER
I20260929 15:58:08.842080 31235 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } }
I20260929 15:58:08.843043 31238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:08.843374 31235 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:08.843528 31238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:08.852138 31051 heartbeater.cc:503] Master 127.23.222.126:42641 was elected leader, sending a full tablet report...
I20260929 15:58:08.859828 31249 catalog_manager.cc:1259] Loaded cluster ID: 0bb7066fd387453e8024b8fb435387d6
I20260929 15:58:08.859962 31249 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: loading cluster ID for follower catalog manager: success
I20260929 15:58:08.862507 31249 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: acquiring CA information for follower catalog manager: success
I20260929 15:58:08.864809 31249 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 15:58:08.978447 31156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:08.978888 31156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:08.979051 31156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:08.986934 31156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:08.987104 31156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:08.987211 31156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:08.987346 31156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:08.987437 31156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:08.997766 31156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:42641,127.23.222.125:43047,127.23.222.124:41413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.124.keytab
--principal=oryx/127.23.222.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.124:41413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.222.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:08.999405 31156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:09.000783 31156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:09.014962 31254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:09.014999 31255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:09.015113 31257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:09.015652 31156 server_base.cc:1034] running on GCE node
I20260929 15:58:09.016628 31156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:09.018463 31156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:09.019717 31156 hybrid_clock.cc:648] HybridClock initialized: now 1790697489019661 us; error 45 us; skew 500 ppm
I20260929 15:58:09.020149 31156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:09 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697489, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:09.023804 31156 init.cc:377] Logged in from keytab as oryx/127.23.222.124@KRBTEST.COM (short username oryx)
I20260929 15:58:09.025910 31156 webserver.cc:460] Webserver started at http://127.23.222.124:39821/ using document root <none> and password file <none>
I20260929 15:58:09.026680 31156 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:09.026829 31156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:09.027164 31156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:09.030555 31156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data/instance:
uuid: "4c34296ad5394e1c8fa12dea09f3d9eb"
format_stamp: "Formatted at 2026-09-29 15:58:09 on dist-test-slave-cq88"
server_key: "e019d3e15fd47ac6e76919cb81b3222c"
server_key_iv: "c75b0cae9727bdab576d04839f964483"
server_key_version: "encryptionkey@0"
I20260929 15:58:09.031500 31156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal/instance:
uuid: "4c34296ad5394e1c8fa12dea09f3d9eb"
format_stamp: "Formatted at 2026-09-29 15:58:09 on dist-test-slave-cq88"
server_key: "e019d3e15fd47ac6e76919cb81b3222c"
server_key_iv: "c75b0cae9727bdab576d04839f964483"
server_key_version: "encryptionkey@0"
I20260929 15:58:09.037398 31156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:58:09.041374 31264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:09.042856 31156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:09.043046 31156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal
uuid: "4c34296ad5394e1c8fa12dea09f3d9eb"
format_stamp: "Formatted at 2026-09-29 15:58:09 on dist-test-slave-cq88"
server_key: "e019d3e15fd47ac6e76919cb81b3222c"
server_key_iv: "c75b0cae9727bdab576d04839f964483"
server_key_version: "encryptionkey@0"
I20260929 15:58:09.043367 31156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:09.063656 31156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:09.067966 31156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:09.068261 31156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:09.087598 31156 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.124:41413
I20260929 15:58:09.087632 31315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.124:41413 every 8 connection(s)
I20260929 15:58:09.089048 31156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
I20260929 15:58:09.091009 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 31156
I20260929 15:58:09.093874 31316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:09.102625 31316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:09 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.124@KRBTEST.COM
Sep 29 15:58:09 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697489, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.124@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:09.116274 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.093322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.23.222.124:41413)
0929 15:58:09.094164 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.094180 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.094224 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:58:09.095929 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.095952 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.096354 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.096862 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.098786 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.100387 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.102184 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.102881 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.103051 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.108298 (+ 5247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.108305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.108315 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.108359 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.111987 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.112654 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.112666 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.112672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.112836 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.113239 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.113245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.113248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.113865 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.114175 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.114653 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.115142 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":371,"threads_started":1}
I20260929 15:58:09.121134 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.105563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.23.222.126:42641)
0929 15:58:09.105731 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.105735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.106428 (+ 693us) server_negotiation.cc:407] Connection header received
0929 15:58:09.106648 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.106654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.106738 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.106855 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.108138 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.108913 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.111228 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.111666 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.111766 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.115424 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.115429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.115432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.115467 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.118341 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.119065 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.119071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.119075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.119284 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.119662 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.119669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.119672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.119958 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.120120 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.120556 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.120783 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:58:09.121192 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.105537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:33662)
0929 15:58:09.106291 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.106326 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.106513 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.106931 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.106938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.107600 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:09.107963 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.107974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.109151 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.109158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.111069 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.111084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.111863 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.111871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.112056 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.112699 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:09.112738 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:09.115200 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:09.118487 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.118500 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.118507 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.118940 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.119405 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.119412 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.119416 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.119556 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.120088 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:09.120095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:09.120278 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.120465 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.120782 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":418,"threads_started":1}
I20260929 15:58:09.122555 31316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:09 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697489, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.124@KRBTEST.COM for oryx/127.23.222.125@KRBTEST.COM
I20260929 15:58:09.135964 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.123178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.0.0.1:39276)
0929 15:58:09.123436 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.123454 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.123535 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.124369 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.124374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.124728 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:09.124882 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.124890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.126154 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.126161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.127102 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.127113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.127861 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.127867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.128046 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.128590 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:09.128607 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:09.130662 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:09.133587 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.133594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.133598 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.133974 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.134350 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.134356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.134359 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.134492 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.134998 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:09.135003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:09.135143 (+ 140us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.135313 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.135619 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 15:58:09.136018 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.123487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.23.222.125:43047)
0929 15:58:09.124059 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.124064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.124086 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:58:09.124134 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.124138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.124194 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.124341 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.125055 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.126021 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.127278 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.127688 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.127824 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.130884 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.130890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.130893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.130925 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.133437 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.134098 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.134110 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.134113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.134231 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.134598 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.134604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.134607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.134871 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.135032 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.135410 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.135619 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":298,"threads_started":1}
I20260929 15:58:09.137107 31316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 15:58:09 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697489, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.124@KRBTEST.COM for oryx/127.23.222.124@KRBTEST.COM
I20260929 15:58:09.149768 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.137942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.23.222.124:41413)
0929 15:58:09.138092 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.138096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.138111 (+ 15us) server_negotiation.cc:407] Connection header received
0929 15:58:09.138156 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.138160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.138208 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.138281 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.138898 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.139728 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.140867 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.141271 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.141442 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.144534 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.144539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.144542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.144570 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.147087 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.147858 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.147861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.147863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.148044 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.148476 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.148479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.148481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.148746 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.148910 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.149269 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.149482 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":54}
I20260929 15:58:09.150029 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.137757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:41413 (local address 127.0.0.1:37792)
0929 15:58:09.137930 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.137945 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.138023 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.138315 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.138318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.138633 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:09.138764 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.138771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.139840 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.139845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.140731 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.140742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.141454 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.141458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.141650 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.142261 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:09.142278 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:09.144345 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:09.147374 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.147378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.147380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.147756 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.148157 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.148160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.148163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.148355 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.148916 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:09.148919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:09.149062 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.149171 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.149647 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":99}
I20260929 15:58:09.155090 31316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: Bootstrap starting.
I20260929 15:58:09.158234 31316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:09.159222 31316 log.cc:824] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:09.161934 31316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: No bootstrap required, opened a new log
I20260929 15:58:09.169574 31316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } }
I20260929 15:58:09.170058 31316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:09.170181 31316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c34296ad5394e1c8fa12dea09f3d9eb, State: Initialized, Role: FOLLOWER
I20260929 15:58:09.170864 31316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } }
I20260929 15:58:09.171797 31323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } } }
I20260929 15:58:09.172201 31323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:09.173005 31316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:09.173791 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:09.174058 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.189390 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.174030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.0.0.1:39278)
0929 15:58:09.181612 (+ 7582us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.181628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.181825 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.182816 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.182821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.182851 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:09.183182 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.183200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.184332 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.184340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.186541 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.186554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.188180 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.188190 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.188473 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.188565 (+ 92us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.188688 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.188910 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7488,"thread_start_us":7378,"threads_started":1}
I20260929 15:58:09.190321 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.174562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.23.222.125:43047)
0929 15:58:09.180169 (+ 5607us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.180179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.181737 (+ 1558us) server_negotiation.cc:407] Connection header received
0929 15:58:09.182485 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.182502 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.182649 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.182778 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:09.183392 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.184129 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.186699 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.187976 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.188168 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.188370 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.189745 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.189970 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5420}
I20260929 15:58:09.191586 31238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d82bc0a64f455b9e4c9bacf9b78ae0. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:09.191844 31238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:09.192160 31326 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: attempting to promote NON_VOTER ba8d5e94f773430f962bb7b2dfc4633f to VOTER
I20260929 15:58:09.193089 31326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } }
W20260929 15:58:09.193439 31339 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 15:58:09.193583 31339 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 15:58:09.195155 31210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 LEARNER]: Refusing update from remote peer 78d82bc0a64f455b9e4c9bacf9b78ae0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 15:58:09.195513 31156 master_runner.cc:428] Detected that this master 4c34296ad5394e1c8fa12dea09f3d9eb is joining an existing cluster
I20260929 15:58:09.195637 31156 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.222.124:41413
I20260929 15:58:09.195715 31324 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 15:58:09.199213 31325 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ba8d5e94f773430f962bb7b2dfc4633f (127.23.222.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } }
I20260929 15:58:09.200310 31210 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ba8d5e94f773430f962bb7b2dfc4633f (127.23.222.125) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } }
I20260929 15:58:09.203542 31326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } }
I20260929 15:58:09.203490 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } }
I20260929 15:58:09.203758 31326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.203781 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.206730 31238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } }
I20260929 15:58:09.207007 31238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:09.224704 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 30656
I20260929 15:58:09.232764 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.211954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.0.0.1:39282)
0929 15:58:09.212986 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.213008 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.213197 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.213762 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.213766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.214448 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:09.214810 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.214833 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.216328 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.216337 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.217987 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.218001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.219224 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.219265 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.219493 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.220104 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:09.220132 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:09.221229 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:09.226675 (+ 5446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.226680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.226684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.229208 (+ 2524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.230478 (+ 1270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.230482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.230485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.230683 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.231519 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:09.231528 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:09.231735 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.231891 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.232220 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":600,"threads_started":1}
I20260929 15:58:09.234243 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.212197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.23.222.125:43047)
0929 15:58:09.212397 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.212403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.213093 (+ 690us) server_negotiation.cc:407] Connection header received
0929 15:58:09.213344 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.213350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.213438 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.213509 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.214997 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.216132 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.218156 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.218781 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.218906 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.221425 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.221431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.221436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.221477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.226470 (+ 4993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.230150 (+ 3680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.230156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.230159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.230338 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.230848 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.230853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.230856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.231269 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.231452 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.233626 (+ 2174us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.233880 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:58:09.236966 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.210859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.23.222.126:42641)
0929 15:58:09.211028 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.211033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.211782 (+ 749us) server_negotiation.cc:407] Connection header received
0929 15:58:09.212035 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.212044 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.212133 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.212208 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.213926 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.215096 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.217316 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.218031 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.218134 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.220994 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.221002 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.221008 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.221054 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.226896 (+ 5842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.228010 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.228018 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.228024 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.228235 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.234944 (+ 6709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.234951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.234954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.235369 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.235548 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.236194 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.236482 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 15:58:09.238279 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.210988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:33666)
0929 15:58:09.211655 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.211671 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.211858 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.212458 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.212464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.213310 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:09.213740 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.213752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.215333 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.215338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.217141 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.217156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.218461 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.218467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.218662 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.219447 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:09.219474 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:09.220760 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:09.227100 (+ 6340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.227108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.227115 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.227822 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.228418 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.228426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.228432 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.234801 (+ 6369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.235628 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:09.235637 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:09.235855 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.237462 (+ 1607us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.237804 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":405,"threads_started":1}
I20260929 15:58:09.246268 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.213126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.23.222.124:41413)
0929 15:58:09.213312 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.213318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.229934 (+ 16616us) server_negotiation.cc:407] Connection header received
0929 15:58:09.230120 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.230125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.230190 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.230276 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.231725 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.232844 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.234741 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.235434 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.235541 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.238349 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.238357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.238363 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.238406 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.242290 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.243077 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.243090 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.243095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.243274 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.243694 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.243698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.243702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.244191 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.245542 (+ 1351us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.245674 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.245922 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:58:09.246832 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.213425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:41413 (local address 127.0.0.1:37802)
0929 15:58:09.229823 (+ 16398us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.229839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.229998 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.230448 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.230454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.231142 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:09.231568 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.231580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.233044 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.233051 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.234584 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.234597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.235808 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.235815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.236001 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.236710 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:09.236734 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:09.238139 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:09.242438 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.242443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.242447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.242948 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.243411 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.243415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.243418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.243569 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.244614 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:09.244619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:09.244805 (+ 186us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.244973 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.245254 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16277,"thread_start_us":16122,"threads_started":1}
Sep 29 15:58:09 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.65@KRBTEST.COM
Sep 29 15:58:09 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697486, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.126@KRBTEST.COM for oryx/127.23.222.124@KRBTEST.COM
I20260929 15:58:09.259909 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.226581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.23.222.65:33817)
0929 15:58:09.239197 (+ 12616us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.239209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.239268 (+ 59us) server_negotiation.cc:407] Connection header received
0929 15:58:09.239349 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.239359 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.239697 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.241765 (+ 2068us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.242024 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.243618 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.245658 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.246809 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.247812 (+ 1003us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.252524 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.252537 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.252552 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.252613 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.256321 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.257229 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.257238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.257242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.257408 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.257877 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.257883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.257900 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.258394 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.258697 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.259106 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.259448 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12406,"thread_start_us":12223,"threads_started":1}
I20260929 15:58:09.261955 30779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:09.262661 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.246311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:41413 (local address 127.0.0.1:37804)
0929 15:58:09.246692 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.246714 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.246818 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.247636 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.247642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.248142 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:09.248325 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.248337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.249991 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.249997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.251224 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.251287 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.252023 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.252029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.252208 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.252803 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:09.252831 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:09.255651 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:09.259852 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.259858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.259863 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.260445 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.260899 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.260904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.260908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.261056 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.261596 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:09.261601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:09.261788 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.261949 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.262303 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260929 15:58:09.262666 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.247036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.23.222.124:41413)
0929 15:58:09.247217 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.247223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.247263 (+ 40us) server_negotiation.cc:407] Connection header received
0929 15:58:09.247357 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.247362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.247469 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.248468 (+ 999us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.248647 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.249805 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.251452 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.251863 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.252010 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.255877 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.255884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.255897 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.255940 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.259686 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.260614 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.260619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.260623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.260762 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.261173 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.261178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.261182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.261467 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.261625 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.262069 (+ 444us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.262303 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:58:09.264540 30594 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.222.124:41413: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb"
member_type: NON_VOTER
last_known_addr {
host: "127.23.222.124"
port: 41413
}
attrs {
promote: true
}
}
dest_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0"
cas_config_opid_index: 6
I20260929 15:58:09.265265 30594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } }
I20260929 15:58:09.267637 31210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Refusing update from remote peer 78d82bc0a64f455b9e4c9bacf9b78ae0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 15:58:09.267977 31291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:09.270655 31291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 FOLLOWER]: Refusing update from remote peer 78d82bc0a64f455b9e4c9bacf9b78ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 15:58:09.271554 31326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 15:58:09.272209 31326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:09.274932 31324 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4c34296ad5394e1c8fa12dea09f3d9eb (127.23.222.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } } }
I20260929 15:58:09.288482 31210 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4c34296ad5394e1c8fa12dea09f3d9eb (127.23.222.124) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } } }
I20260929 15:58:09.299975 31238 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } } }
I20260929 15:58:09.300307 31238 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:09.300593 31324 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.222.124:41413
I20260929 15:58:09.301292 31156 master.cc:537] Master@127.23.222.124:41413 shutting down...
I20260929 15:58:09.301285 31325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } } }
I20260929 15:58:09.301584 31325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.302796 31325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } } }
I20260929 15:58:09.303084 31325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.304450 31291 raft_consensus.cc:858] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } }
W20260929 15:58:09.304915 31291 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } }
I20260929 15:58:09.346412 31291 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ba8d5e94f773430f962bb7b2dfc4633f (127.23.222.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:09.350348 31356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:09.350870 31356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 15:58:09.350919 31323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d82bc0a64f455b9e4c9bacf9b78ae0. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:09.351198 31323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 15:58:09.351128 31353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: NON_VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: true } } }
I20260929 15:58:09.351804 31353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260929 15:58:09.353204 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } } }
I20260929 15:58:09.353577 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
W20260929 15:58:09.354468 30580 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 15:58:09.355198 31156 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260929 15:58:09.355736 31156 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260929 15:58:09.355953 31156 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: stopping tablet replica
W20260929 15:58:09.357602 30580 consensus_peers.cc:487] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 -> Peer 4c34296ad5394e1c8fa12dea09f3d9eb (127.23.222.124:41413): Couldn't send request to peer 4c34296ad5394e1c8fa12dea09f3d9eb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 15:58:09.374369 31156 master.cc:552] Master@127.23.222.124:41413 shutdown complete.
I20260929 15:58:09.374485 31156 master_runner.cc:305] Clearing existing system tablet
I20260929 15:58:09.374836 31156 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:09.374972 31156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:09.377504 31156 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260929 15:58:09.379079 31359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:09.379537 31156 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:09.379796 31156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 15:58:09.379894 31156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal
uuid: "4c34296ad5394e1c8fa12dea09f3d9eb"
format_stamp: "Formatted at 2026-09-29 15:58:09 on dist-test-slave-cq88"
server_key: "e019d3e15fd47ac6e76919cb81b3222c"
server_key_iv: "c75b0cae9727bdab576d04839f964483"
server_key_version: "encryptionkey@0"
I20260929 15:58:09.383392 31156 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:09.388417 31156 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 15:58:09.388585 31156 log.cc:1196] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 15:58:09.389101 31156 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: Deleting consensus metadata
I20260929 15:58:09.389358 31156 master_runner.cc:315] Copying system tablet from 127.23.222.126:42641
I20260929 15:58:09.391901 31156 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: tablet copy: Beginning tablet copy session from remote peer at address 127.23.222.126:42641
I20260929 15:58:09.405082 31129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.392667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.23.222.126:42641)
0929 15:58:09.392885 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.392895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.393421 (+ 526us) server_negotiation.cc:407] Connection header received
0929 15:58:09.393654 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.393659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.393742 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.393856 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:09.394685 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.395545 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.396960 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.397373 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.397508 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.399482 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:09.399494 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:09.399497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:09.399532 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:09.402366 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.403038 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.403043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.403047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.403181 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:09.403628 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:09.403634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:09.403637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:09.403926 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:09.404098 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.404482 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.404705 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 15:58:09.405143 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.392647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:33680)
0929 15:58:09.393292 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.393312 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.393486 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.393872 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.393876 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.394310 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:09.394529 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.394538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.395695 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.395700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.396797 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.396809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.397546 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.397553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.397743 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.398404 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:09.398422 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:09.399183 (+ 761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:09.402523 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.402528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.402532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.402906 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.403355 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:09.403360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:09.403364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:09.403515 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:09.404048 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:09.404053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:09.404226 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.404377 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.404719 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":398,"threads_started":1}
I20260929 15:58:09.406087 30615 tablet_copy_service.cc:140] P 78d82bc0a64f455b9e4c9bacf9b78ae0: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4c34296ad5394e1c8fa12dea09f3d9eb ({username='oryx', principal='oryx/127.23.222.124@KRBTEST.COM'} at 127.0.0.1:33680)
I20260929 15:58:09.406262 30615 tablet_copy_service.cc:161] P 78d82bc0a64f455b9e4c9bacf9b78ae0: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4c34296ad5394e1c8fa12dea09f3d9eb at {username='oryx', principal='oryx/127.23.222.124@KRBTEST.COM'} at 127.0.0.1:33680: session id = 4c34296ad5394e1c8fa12dea09f3d9eb-00000000000000000000000000000000
I20260929 15:58:09.409014 30615 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 15:58:09.410733 31156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 15:58:09.415719 31156 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: tablet copy: Starting download of 0 data blocks...
I20260929 15:58:09.416038 31156 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: tablet copy: Starting download of 1 WAL segments...
I20260929 15:58:09.418622 31156 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 15:58:09.421980 30615 tablet_copy_service.cc:342] P 78d82bc0a64f455b9e4c9bacf9b78ae0: Request end of tablet copy session 4c34296ad5394e1c8fa12dea09f3d9eb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.222.124@KRBTEST.COM'} at 127.0.0.1:33680
I20260929 15:58:09.422128 30615 tablet_copy_service.cc:433] P 78d82bc0a64f455b9e4c9bacf9b78ae0: ending tablet copy session 4c34296ad5394e1c8fa12dea09f3d9eb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4c34296ad5394e1c8fa12dea09f3d9eb
I20260929 15:58:09.433665 31156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:09.436347 31366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:09.440743 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 30656
W20260929 15:58:09.443483 31367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:09.459668 31156 server_base.cc:1034] running on GCE node
W20260929 15:58:09.461081 31369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:09.461587 31156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:09.462293 31156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:09.463513 31156 hybrid_clock.cc:648] HybridClock initialized: now 1790697489463451 us; error 52 us; skew 500 ppm
I20260929 15:58:09.463785 31156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:09.465652 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:33817
--local_ip_for_outbound_sockets=127.23.222.65
--tserver_master_addrs=127.23.222.126:42641
--webserver_port=39833
--webserver_interface=127.23.222.65
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab
--principal=oryx/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.222.126:42641,127.23.222.125:43047
--tserver_master_addrs=127.23.222.126:42641,127.23.222.125:43047,127.23.222.124:41413 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:58:09 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697489, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:09.466624 31156 init.cc:377] Logged in from keytab as oryx/127.23.222.124@KRBTEST.COM (short username oryx)
I20260929 15:58:09.468286 31156 webserver.cc:460] Webserver started at http://127.23.222.124:41051/ using document root <none> and password file <none>
I20260929 15:58:09.468695 31156 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:09.468796 31156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:09.472206 31156 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260929 15:58:09.474090 31377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:09.474576 31156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 15:58:09.474720 31156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal
uuid: "4c34296ad5394e1c8fa12dea09f3d9eb"
format_stamp: "Formatted at 2026-09-29 15:58:09 on dist-test-slave-cq88"
server_key: "e019d3e15fd47ac6e76919cb81b3222c"
server_key_iv: "c75b0cae9727bdab576d04839f964483"
server_key_version: "encryptionkey@0"
I20260929 15:58:09.474911 31156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:09.501576 31156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:09.505651 31156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:09.525985 31156 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.124:41413
I20260929 15:58:09.526144 31428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.124:41413 every 8 connection(s)
I20260929 15:58:09.527114 31156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
I20260929 15:58:09.532299 31429 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:58:09.536062 31429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: Bootstrap starting.
I20260929 15:58:09.552042 31429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:58:09.552627 31429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: Bootstrap complete.
I20260929 15:58:09.553807 31429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } }
I20260929 15:58:09.554148 31429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4c34296ad5394e1c8fa12dea09f3d9eb, State: Initialized, Role: LEARNER
I20260929 15:58:09.554425 31429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } }
I20260929 15:58:09.555449 31432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } } }
I20260929 15:58:09.555750 31429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:09.555938 31432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:09.565390 31443 catalog_manager.cc:1259] Loaded cluster ID: 0bb7066fd387453e8024b8fb435387d6
I20260929 15:58:09.565511 31443 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: loading cluster ID for follower catalog manager: success
I20260929 15:58:09.569105 31443 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: acquiring CA information for follower catalog manager: success
I20260929 15:58:09.571691 31443 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 15:58:09.849118 31376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:09.849435 31376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:09.849601 31376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:09.858598 31376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:09.858778 31376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:09.859012 31376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:58:09.869170 31376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab
--principal=oryx/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:33817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=39833
--tserver_master_addrs=127.23.222.126:42641,127.23.222.125:43047,127.23.222.124:41413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:09.870590 31376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:09.872048 31376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:09.885946 31449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:09.885982 31448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:09.886435 31451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:09.887928 31376 server_base.cc:1034] running on GCE node
I20260929 15:58:09.888530 31376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:09.889549 31376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:09.890758 31376 hybrid_clock.cc:648] HybridClock initialized: now 1790697489890703 us; error 54 us; skew 500 ppm
I20260929 15:58:09.891088 31376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:09 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697489, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:09.894201 31376 init.cc:377] Logged in from keytab as oryx/127.23.222.65@KRBTEST.COM (short username oryx)
I20260929 15:58:09.895658 31376 webserver.cc:460] Webserver started at http://127.23.222.65:39833/ using document root <none> and password file <none>
I20260929 15:58:09.896319 31376 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:09.896431 31376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:09.901880 31376 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 15:58:09.904664 31458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:09.906479 31376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:09.906643 31376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "83aec33916c44295876e27cc9e18e527"
format_stamp: "Formatted at 2026-09-29 15:58:06 on dist-test-slave-cq88"
server_key: "b162d95edf22a67eb96afea39564d886"
server_key_iv: "8904114750686e2b92d204c803ee28cf"
server_key_version: "encryptionkey@0"
I20260929 15:58:09.907692 31376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:09.931650 31376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:09.936066 31376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:09.936305 31376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:09.937274 31376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:09.938704 31376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:09.938843 31376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:09.938941 31376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:09.939000 31376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:09.953258 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } } }
I20260929 15:58:09.953604 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.967031 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.953425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:41413 (local address 127.0.0.1:37812)
0929 15:58:09.958816 (+ 5391us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:09.958844 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:09.959061 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:09.959764 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:09.959767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:09.959790 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:09.960006 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.960021 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.962302 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.962306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.964137 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:09.964154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.966087 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.966096 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:09.966318 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.966377 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 15:58:09.966479 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 15:58:09.966601 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5233,"thread_start_us":5134,"threads_started":1}
I20260929 15:58:09.967429 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.953946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.23.222.124:41413)
0929 15:58:09.958374 (+ 4428us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.958384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:09.959017 (+ 633us) server_negotiation.cc:407] Connection header received
0929 15:58:09.959265 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:09.959275 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:09.959401 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:09.959544 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:09.960206 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.962108 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.964281 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:09.965559 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:09.965794 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:09.965973 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:09.966779 (+ 806us) server_negotiation.cc:299] Negotiation successful
0929 15:58:09.966944 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4152,"thread_start_us":4011,"threads_started":1}
I20260929 15:58:09.968291 31404 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260929 15:58:09.968757 31432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d82bc0a64f455b9e4c9bacf9b78ae0. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: NON_VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: true } } }
I20260929 15:58:09.969036 31432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: This master's current role is: LEARNER
I20260929 15:58:09.969470 31324 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: attempting to promote NON_VOTER 4c34296ad5394e1c8fa12dea09f3d9eb to VOTER
I20260929 15:58:09.970355 31324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } }
I20260929 15:58:09.972461 31210 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Refusing update from remote peer 78d82bc0a64f455b9e4c9bacf9b78ae0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 15:58:09.972635 31404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 LEARNER]: Refusing update from remote peer 78d82bc0a64f455b9e4c9bacf9b78ae0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 15:58:09.973145 31505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 15:58:09.973915 31503 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 15:58:09.975939 31505 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4c34296ad5394e1c8fa12dea09f3d9eb (127.23.222.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:09.979683 31503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:09.979977 31503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.980271 31503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:09.980512 31503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.980044 31210 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4c34296ad5394e1c8fa12dea09f3d9eb (127.23.222.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:09.980782 31503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:09.981002 31503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:09.980628 31404 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4c34296ad5394e1c8fa12dea09f3d9eb (127.23.222.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:09.983760 31432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:09.984020 31432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:09.985726 31551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:09.986187 31551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:09.994760 31376 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:33817
I20260929 15:58:09.994849 31577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:33817 every 8 connection(s)
I20260929 15:58:09.996243 31376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:10.002964 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 31376
I20260929 15:58:10.003334 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 30790
Sep 29 15:58:10 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697489, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.125@KRBTEST.COM
Sep 29 15:58:10 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697489, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
Sep 29 15:58:10 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697489, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.124@KRBTEST.COM
I20260929 15:58:10.030320 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.001365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:40501 (local address 127.23.222.126:42641)
0929 15:58:10.002460 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:10.002466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.002489 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:58:10.003967 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.003973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.004070 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.004178 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:10.007696 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.009390 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.013600 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.014261 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.014363 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.021785 (+ 7422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:10.021792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:10.021796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:10.021838 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:10.026155 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.027211 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.027216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.027220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.027426 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.028007 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.028013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.028019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.028399 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:10.028538 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:10.029621 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0929 15:58:10.029862 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":973,"thread_start_us":814,"threads_started":1}
I20260929 15:58:10.033147 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.001231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.65:40501)
0929 15:58:10.001899 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:10.001919 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:10.003530 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:10.004661 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:10.004678 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:10.005869 (+ 1191us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:10.007383 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.007416 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.009974 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.009979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.013414 (+ 3435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.013427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.014761 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.014768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.015123 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.016312 (+ 1189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:10.016343 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:10.021482 (+ 5139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:10.026351 (+ 4869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.026363 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.026404 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.027065 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.027589 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.027593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.027596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.027867 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.028637 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:10.028648 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:10.029140 (+ 492us) client_negotiation.cc:769] Sending connection context
0929 15:58:10.030638 (+ 1498us) client_negotiation.cc:240] Negotiation successful
0929 15:58:10.031394 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":319,"spinlock_wait_cycles":31488,"thread_start_us":323,"threads_started":1}
I20260929 15:58:10.034027 31508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.002203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:33629 (local address 127.23.222.124:41413)
0929 15:58:10.002432 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:10.002440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.003410 (+ 970us) server_negotiation.cc:407] Connection header received
0929 15:58:10.009883 (+ 6473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.009891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.009984 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.010068 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:10.013429 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.014553 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.018860 (+ 4307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.019772 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.019937 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.026781 (+ 6844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:10.026791 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:10.026796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:10.026843 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:10.030449 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.031506 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.031512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.031516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.031673 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.032207 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.032212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.032216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.032609 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:10.032750 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:10.033380 (+ 630us) server_negotiation.cc:299] Negotiation successful
0929 15:58:10.033657 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 15:58:10.034521 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.999354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:54235 (local address 127.23.222.125:43047)
0929 15:58:09.999908 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:09.999914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.000814 (+ 900us) server_negotiation.cc:407] Connection header received
0929 15:58:10.003900 (+ 3086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.003908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.003997 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.004108 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:10.007678 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.009354 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.013668 (+ 4314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.014606 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.014732 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.023324 (+ 8592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:10.023331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:10.023336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:10.023380 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:10.027097 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.031561 (+ 4464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.031567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.031570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.031742 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.032235 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.032239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.032242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.032576 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:10.032733 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:10.033658 (+ 925us) server_negotiation.cc:299] Negotiation successful
0929 15:58:10.033926 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":233,"threads_started":1}
I20260929 15:58:10.034948 31579 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42641
Sep 29 15:58:10 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.66@KRBTEST.COM
I20260929 15:58:10.035094 31586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.002572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:41413 (local address 127.23.222.65:33629)
0929 15:58:10.003293 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:10.003311 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:10.003545 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:10.010327 (+ 6782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:10.010334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:10.012097 (+ 1763us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:10.012688 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.012702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.014748 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.014753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.016934 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.016956 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.020252 (+ 3296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.020262 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.020568 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.021456 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:10.021489 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:10.026522 (+ 5033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:10.030573 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.030578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.030582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.031345 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.031848 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.031852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.031856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.032064 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.032856 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:10.032861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:10.033053 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 15:58:10.034294 (+ 1241us) client_negotiation.cc:240] Negotiation successful
0929 15:58:10.034636 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":338,"threads_started":1}
I20260929 15:58:10.035327 31579 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:10.036064 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:09.999903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.23.222.65:54235)
0929 15:58:10.000648 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:10.000686 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:10.003525 (+ 2839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:10.004660 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:10.004678 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:10.005887 (+ 1209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:10.007379 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.007416 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.010528 (+ 3112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.010534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.013413 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.013435 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.015196 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.015205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.015592 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.016317 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:10.016369 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:10.023028 (+ 6659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:10.027662 (+ 4634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.027669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.027674 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.031408 (+ 3734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.031915 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.031920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.031924 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.032100 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.032816 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:10.032827 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:10.033031 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 15:58:10.034953 (+ 1922us) client_negotiation.cc:240] Negotiation successful
0929 15:58:10.035613 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":301,"spinlock_wait_cycles":9856,"thread_start_us":297,"threads_started":1}
I20260929 15:58:10.036329 31579 heartbeater.cc:511] Master 127.23.222.126:42641 requested a full tablet report, sending...
I20260929 15:58:10.037586 31580 heartbeater.cc:347] Connected to a master server at 127.23.222.125:43047
I20260929 15:58:10.037791 31580 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:10.038254 31580 heartbeater.cc:511] Master 127.23.222.125:43047 requested a full tablet report, sending...
I20260929 15:58:10.037611 31578 heartbeater.cc:347] Connected to a master server at 127.23.222.124:41413
I20260929 15:58:10.038590 31578 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:10.038563 30594 ts_manager.cc:194] Re-registered known tserver with Master: 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817)
I20260929 15:58:10.038957 31578 heartbeater.cc:511] Master 127.23.222.124:41413 requested a full tablet report, sending...
I20260929 15:58:10.039793 30594 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:40501
I20260929 15:58:10.040807 31200 ts_manager.cc:194] Registered new tserver with Master: 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817)
I20260929 15:58:10.041343 31394 ts_manager.cc:194] Registered new tserver with Master: 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817)
I20260929 15:58:10.043740 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.004372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.23.222.66:38077)
0929 15:58:10.022998 (+ 18626us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:10.023008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.023044 (+ 36us) server_negotiation.cc:407] Connection header received
0929 15:58:10.023120 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.023128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.023457 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.023651 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:10.026374 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.027933 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.029491 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.030161 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.030318 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.036274 (+ 5956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:10.036288 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:10.036300 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:10.036337 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:10.040279 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.041184 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.041193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.041197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.041371 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.041972 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.041977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.041980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.042428 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:10.042652 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:10.043038 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 15:58:10.043229 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18413,"thread_start_us":18233,"threads_started":1}
I20260929 15:58:10.044935 30913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:10.128960 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 30790
I20260929 15:58:10.145821 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:38077
--local_ip_for_outbound_sockets=127.23.222.66
--tserver_master_addrs=127.23.222.126:42641
--webserver_port=44421
--webserver_interface=127.23.222.66
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab
--principal=oryx/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.222.126:42641,127.23.222.125:43047
--tserver_master_addrs=127.23.222.126:42641,127.23.222.125:43047,127.23.222.124:41413 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:10.487035 31590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:10.487432 31590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:10.487600 31590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:10.494975 31590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:10.495143 31590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:10.495368 31590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:58:10.505386 31590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab
--principal=oryx/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:38077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=44421
--tserver_master_addrs=127.23.222.126:42641,127.23.222.125:43047,127.23.222.124:41413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:10.506844 31590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:10.508287 31590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:10.522380 31595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:10.522645 31598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:10.522697 31596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:10.523298 31590 server_base.cc:1034] running on GCE node
I20260929 15:58:10.524026 31590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:10.525130 31590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:10.526377 31590 hybrid_clock.cc:648] HybridClock initialized: now 1790697490526327 us; error 43 us; skew 500 ppm
I20260929 15:58:10.526885 31590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:10 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697490, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:10.530342 31590 init.cc:377] Logged in from keytab as oryx/127.23.222.66@KRBTEST.COM (short username oryx)
I20260929 15:58:10.536024 31590 webserver.cc:460] Webserver started at http://127.23.222.66:44421/ using document root <none> and password file <none>
I20260929 15:58:10.536893 31590 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:10.537062 31590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:10.542900 31590 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:58:10.546159 31605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:10.547876 31590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:58:10.548070 31590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "f3f02f25c6ca4061902c1fb1a305d7c6"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-cq88"
server_key: "d543414bfdb050714b6169048aff1821"
server_key_iv: "c23e2b11f83a109202e2b6693d7bf00c"
server_key_version: "encryptionkey@0"
I20260929 15:58:10.549062 31590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:10.578670 31590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:10.582921 31590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:10.583261 31590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:10.584273 31590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:10.585749 31590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:10.585870 31590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:10.585968 31590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:10.586053 31590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:10.619508 31590 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:38077
I20260929 15:58:10.619596 31717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:38077 every 8 connection(s)
I20260929 15:58:10.620976 31590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:10.627450 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 31590
I20260929 15:58:10.627763 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 30923
Sep 29 15:58:10 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697490, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.125@KRBTEST.COM
Sep 29 15:58:10 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697490, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.124@KRBTEST.COM
I20260929 15:58:10.654201 31722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.624044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:38179 (local address 127.23.222.125:43047)
0929 15:58:10.624467 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:10.624472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.625626 (+ 1154us) server_negotiation.cc:407] Connection header received
0929 15:58:10.629177 (+ 3551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.629187 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.629305 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.629423 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:10.632489 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.634040 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.637550 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.638458 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.638577 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.645288 (+ 6711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:10.645299 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:10.645306 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:10.645365 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:10.649148 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.650181 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.650189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.650193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.650644 (+ 451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.651299 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.651325 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.651328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.651705 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:10.651851 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:10.653546 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0929 15:58:10.653807 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":219,"threads_started":1}
I20260929 15:58:10.657394 31723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.624659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.23.222.66:38179)
0929 15:58:10.625468 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:10.625515 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:10.628746 (+ 3231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:10.630111 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:10.630128 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:10.631128 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:10.632209 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.632241 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.634317 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.634325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.637275 (+ 2950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.637318 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.638953 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.638959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.639333 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.640461 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:10.640506 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:10.644912 (+ 4406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:10.649333 (+ 4421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.649344 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.649364 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.650026 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.650814 (+ 788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.650820 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.650824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.651123 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.651954 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:10.651969 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:10.653027 (+ 1058us) client_negotiation.cc:769] Sending connection context
0929 15:58:10.654738 (+ 1711us) client_negotiation.cc:240] Negotiation successful
0929 15:58:10.655370 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":338,"threads_started":1}
I20260929 15:58:10.659138 31720 heartbeater.cc:347] Connected to a master server at 127.23.222.125:43047
I20260929 15:58:10.659603 31720 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:10.660805 31720 heartbeater.cc:511] Master 127.23.222.125:43047 requested a full tablet report, sending...
I20260929 15:58:10.661386 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.626265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:47547 (local address 127.23.222.124:41413)
0929 15:58:10.626694 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:10.626698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.638939 (+ 12241us) server_negotiation.cc:407] Connection header received
0929 15:58:10.639102 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.639112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.639231 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.639397 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:10.643823 (+ 4426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.645435 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.648726 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.649417 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.649526 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.653673 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:10.653679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:10.653683 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:10.653727 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:10.657325 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.658769 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.658774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.658778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.658937 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.659430 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.659437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.659444 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.659809 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:10.659964 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:10.660694 (+ 730us) server_negotiation.cc:299] Negotiation successful
0929 15:58:10.660991 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":244,"threads_started":1}
I20260929 15:58:10.662552 31200 ts_manager.cc:194] Registered new tserver with Master: f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077)
I20260929 15:58:10.664330 31724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.626050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:41413 (local address 127.23.222.66:47547)
0929 15:58:10.638811 (+ 12761us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:10.638832 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:10.638966 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:10.640183 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:10.640193 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:10.642054 (+ 1861us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:10.642571 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.642588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.645781 (+ 3193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.645790 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.648530 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.648547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.649866 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.649874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.650041 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.650644 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:10.650666 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:10.653448 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:10.657498 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.657502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.657505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.658636 (+ 1131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.659077 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.659081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.659085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.659294 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.659926 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:10.659931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:10.660090 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 15:58:10.661626 (+ 1536us) client_negotiation.cc:240] Negotiation successful
0929 15:58:10.661943 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12590,"thread_start_us":12997,"threads_started":1}
I20260929 15:58:10.665158 31718 heartbeater.cc:347] Connected to a master server at 127.23.222.124:41413
I20260929 15:58:10.665301 31718 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:10.665771 31718 heartbeater.cc:511] Master 127.23.222.124:41413 requested a full tablet report, sending...
I20260929 15:58:10.666872 31394 ts_manager.cc:194] Registered new tserver with Master: f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077)
Sep 29 15:58:10 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697490, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
Sep 29 15:58:10 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.67@KRBTEST.COM
I20260929 15:58:10.676367 31728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.639555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.66:54019)
0929 15:58:10.646514 (+ 6959us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:10.646535 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:10.646930 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:10.660275 (+ 13345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:10.660280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:10.662425 (+ 2145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:10.662739 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.662750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.664055 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.664061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.666263 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:10.666280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.667260 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.667267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:10.667449 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.668030 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:10.668052 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:10.670389 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:10.673534 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.673540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.673544 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.673902 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.674391 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:10.674399 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:10.674405 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:10.674619 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:10.675432 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:10.675438 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:10.675633 (+ 195us) client_negotiation.cc:769] Sending connection context
0929 15:58:10.675752 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 15:58:10.676015 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6793,"thread_start_us":7068,"threads_started":1}
I20260929 15:58:10.676587 31729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.639758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:54019 (local address 127.23.222.126:42641)
0929 15:58:10.655721 (+ 15963us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:10.655728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.655751 (+ 23us) server_negotiation.cc:407] Connection header received
0929 15:58:10.660068 (+ 4317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.660074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.660159 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.660264 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:10.662893 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.663903 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.666414 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.667036 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.667153 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.670591 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:10.670596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:10.670599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:10.670632 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:10.673387 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.674023 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.674029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.674032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.674214 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.674823 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.674829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.674832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.675240 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:10.675448 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:10.675920 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 15:58:10.676159 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14876,"thread_start_us":10733,"threads_started":1}
I20260929 15:58:10.677400 31719 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42641
I20260929 15:58:10.677561 31719 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:10.678030 31719 heartbeater.cc:511] Master 127.23.222.126:42641 requested a full tablet report, sending...
I20260929 15:58:10.679118 30594 ts_manager.cc:194] Re-registered known tserver with Master: f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077)
I20260929 15:58:10.680214 30594 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:54019
I20260929 15:58:10.680775 31727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:10.628736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.23.222.67:38233)
0929 15:58:10.660883 (+ 32147us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:10.660892 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:10.661459 (+ 567us) server_negotiation.cc:407] Connection header received
0929 15:58:10.661634 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:10.661644 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:10.661918 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:10.662098 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:10.664270 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.665814 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.667607 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:10.668510 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:10.668648 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:10.673504 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:10.673517 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:10.673542 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:10.673577 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:10.677476 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.678395 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.678402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.678404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.678619 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:10.679127 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:10.679130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:10.679132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:10.679538 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:10.679764 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:10.680124 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 15:58:10.680391 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31951,"thread_start_us":34128,"threads_started":1}
I20260929 15:58:10.682117 31046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:10.765805 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 30923
I20260929 15:58:10.782284 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:38233
--local_ip_for_outbound_sockets=127.23.222.67
--tserver_master_addrs=127.23.222.126:42641
--webserver_port=33067
--webserver_interface=127.23.222.67
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab
--principal=oryx/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.222.126:42641,127.23.222.125:43047
--tserver_master_addrs=127.23.222.126:42641,127.23.222.125:43047,127.23.222.124:41413 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:11.043866 31579 heartbeater.cc:503] Master 127.23.222.126:42641 was elected leader, sending a full tablet report...
W20260929 15:58:11.122141 31731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:11.122421 31731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:11.122514 31731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:11.129745 31731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:11.129859 31731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:11.129978 31731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:58:11.139976 31731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab
--principal=oryx/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:38233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=33067
--tserver_master_addrs=127.23.222.126:42641,127.23.222.125:43047,127.23.222.124:41413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:11.141418 31731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:11.142853 31731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:11.157173 31736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:11.157073 31739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:11.157325 31737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:11.158577 31731 server_base.cc:1034] running on GCE node
I20260929 15:58:11.159446 31731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:11.160496 31731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:11.161746 31731 hybrid_clock.cc:648] HybridClock initialized: now 1790697491161683 us; error 52 us; skew 500 ppm
I20260929 15:58:11.162181 31731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:11 dist-test-slave-cq88 krb5kdc[30550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697491, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:11.165580 31731 init.cc:377] Logged in from keytab as oryx/127.23.222.67@KRBTEST.COM (short username oryx)
I20260929 15:58:11.169948 31731 webserver.cc:460] Webserver started at http://127.23.222.67:33067/ using document root <none> and password file <none>
I20260929 15:58:11.170739 31731 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:11.170900 31731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:11.177058 31731 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:58:11.180601 31746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:11.182163 31731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:11.182376 31731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "495c68ea69d948d2888be1132f244819"
format_stamp: "Formatted at 2026-09-29 15:58:07 on dist-test-slave-cq88"
server_key: "c8fb6b951f96265f1fde032523a38f98"
server_key_iv: "1c099cb0d6010d6632b60ef5f2f3f7be"
server_key_version: "encryptionkey@0"
I20260929 15:58:11.183492 31731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:11.207052 31731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:11.212085 31731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:11.212405 31731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:11.213611 31731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:11.215020 31731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:11.215108 31731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:11.215178 31731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:11.215279 31731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:11.250519 31731 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:38233
I20260929 15:58:11.250643 31858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:38233 every 8 connection(s)
I20260929 15:58:11.252151 31731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:11.254020 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 31731
Sep 29 15:58:11 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697491, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
Sep 29 15:58:11 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697491, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.125@KRBTEST.COM
I20260929 15:58:11.304558 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.258748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.23.222.126:42641)
0929 15:58:11.278889 (+ 20141us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.278902 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.278931 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:58:11.279001 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.279009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.279118 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.279263 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.282895 (+ 3632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.284434 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.290214 (+ 5780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.290937 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.291078 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.295821 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.295828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.295834 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.295872 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.299938 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.301542 (+ 1604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.301548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.301552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.301733 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.302255 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.302263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.302268 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.302692 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.302949 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.303892 (+ 943us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.304124 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19959,"spinlock_wait_cycles":10624,"thread_start_us":18272,"threads_started":1}
Sep 29 15:58:11 dist-test-slave-cq88 krb5kdc[30550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697491, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.124@KRBTEST.COM
I20260929 15:58:11.307356 31863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.255528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:39313 (local address 127.23.222.126:42641)
0929 15:58:11.256035 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.256043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.257887 (+ 1844us) server_negotiation.cc:407] Connection header received
0929 15:58:11.270555 (+ 12668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.270566 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.270690 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.270836 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.274577 (+ 3741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.276135 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.287577 (+ 11442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.288488 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.288666 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.295841 (+ 7175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.295850 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.295855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.295917 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.302788 (+ 6871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.304017 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.304022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.304028 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.304157 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.304969 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.304973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.304977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.305444 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.305622 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.306594 (+ 972us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.306875 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"spinlock_wait_cycles":896,"thread_start_us":244,"threads_started":1}
I20260929 15:58:11.308686 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.256799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.67:39313)
0929 15:58:11.257687 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:11.257742 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:11.270067 (+ 12325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:11.272084 (+ 2017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:11.272097 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:11.273163 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:11.274314 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.274349 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.284712 (+ 10363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.284721 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.287359 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.287382 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.291947 (+ 4565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.291957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.292133 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.292753 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:11.292779 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:11.295523 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:11.302914 (+ 7391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:11.302924 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:11.302947 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:11.303725 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:11.304272 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:11.304279 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:11.304286 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:11.304568 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:11.305613 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:11.305622 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:11.306146 (+ 524us) client_negotiation.cc:769] Sending connection context
0929 15:58:11.306497 (+ 351us) client_negotiation.cc:240] Negotiation successful
0929 15:58:11.307107 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":362,"threads_started":1}
I20260929 15:58:11.310545 31860 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42641
I20260929 15:58:11.310926 31860 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:11.310968 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.276551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.23.222.125:43047)
0929 15:58:11.283166 (+ 6615us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.283175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.283200 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:58:11.283305 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.283313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.283424 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.283570 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.286743 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.289896 (+ 3153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.291907 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.292610 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.292748 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.296557 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.296565 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.296572 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.296616 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.303212 (+ 6596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.305672 (+ 2460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.305681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.305688 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.305870 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.306380 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.306385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.306389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.308223 (+ 1834us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.308486 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.310256 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.310494 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6422,"thread_start_us":9868,"threads_started":1}
I20260929 15:58:11.311965 31860 heartbeater.cc:511] Master 127.23.222.126:42641 requested a full tablet report, sending...
I20260929 15:58:11.313725 30595 ts_manager.cc:194] Re-registered known tserver with Master: 495c68ea69d948d2888be1132f244819 (127.23.222.67:38233)
I20260929 15:58:11.315016 30595 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:39313
I20260929 15:58:11.316026 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.258466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:55403 (local address 127.23.222.125:43047)
0929 15:58:11.276503 (+ 18037us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.276513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.276539 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:58:11.276614 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.276622 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.276730 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.276889 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.279556 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.280960 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.289033 (+ 8073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.289899 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.290024 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.305931 (+ 15907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.305940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.305945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.305996 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.309611 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.313905 (+ 4294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.313911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.313914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.314070 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.314442 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.314447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.314450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.314742 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.314901 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.315375 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.315592 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17851,"thread_start_us":17847,"threads_started":1}
I20260929 15:58:11.316062 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.258296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.23.222.67:55403)
0929 15:58:11.271080 (+ 12784us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:11.271105 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:11.271273 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:11.277255 (+ 5982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:11.277263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:11.278084 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:11.278508 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.278529 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.281226 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.281236 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.288783 (+ 7547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.288815 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.290791 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.290801 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.295298 (+ 4497us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.296095 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:11.296120 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:11.305634 (+ 9514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:11.313021 (+ 7387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:11.313028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:11.313032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:11.313728 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:11.314195 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:11.314198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:11.314201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:11.314319 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:11.314879 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:11.314884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:11.315043 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 15:58:11.315269 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 15:58:11.315589 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12608,"thread_start_us":13073,"threads_started":1}
I20260929 15:58:11.317261 31861 heartbeater.cc:347] Connected to a master server at 127.23.222.125:43047
I20260929 15:58:11.317435 31861 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:11.317989 31861 heartbeater.cc:511] Master 127.23.222.125:43047 requested a full tablet report, sending...
I20260929 15:58:11.319144 31200 ts_manager.cc:194] Registered new tserver with Master: 495c68ea69d948d2888be1132f244819 (127.23.222.67:38233)
I20260929 15:58:11.313856 30594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:11.324295 30594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:11.328387 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.272156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:51701 (local address 127.23.222.124:41413)
0929 15:58:11.289870 (+ 17714us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.289877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.289898 (+ 21us) server_negotiation.cc:407] Connection header received
0929 15:58:11.289960 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.289968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.290046 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.290149 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.294675 (+ 4526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.295989 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.300528 (+ 4539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.301162 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.301302 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.309429 (+ 8127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.309435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.309439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.309483 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.324774 (+ 15291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.325913 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.325923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.325927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.326111 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.326625 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.326631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.326635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.327056 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.327229 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.327727 (+ 498us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.327967 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17580,"thread_start_us":18924,"threads_started":1}
I20260929 15:58:11.328419 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.278213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:41413 (local address 127.23.222.67:51701)
0929 15:58:11.287590 (+ 9377us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:11.287611 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:11.287723 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:11.290352 (+ 2629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:11.290358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:11.291028 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:11.294093 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.294106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.296169 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.296177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.300345 (+ 4168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.300365 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.302029 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.302036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.302218 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.302801 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:11.302826 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:11.309182 (+ 6356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:11.324992 (+ 15810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:11.324997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:11.325000 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:11.325673 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:11.326264 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:11.326270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:11.326274 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:11.326462 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:11.327197 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:11.327202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:11.327426 (+ 224us) client_negotiation.cc:769] Sending connection context
0929 15:58:11.327586 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 15:58:11.327946 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9227,"thread_start_us":9578,"threads_started":1}
I20260929 15:58:11.329437 31859 heartbeater.cc:347] Connected to a master server at 127.23.222.124:41413
I20260929 15:58:11.329636 31859 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:11.330089 31859 heartbeater.cc:511] Master 127.23.222.124:41413 requested a full tablet report, sending...
I20260929 15:58:11.330906 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.291311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.23.222.124:41413)
0929 15:58:11.291811 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.291819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.291836 (+ 17us) server_negotiation.cc:407] Connection header received
0929 15:58:11.292506 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.292512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.294294 (+ 1782us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.294399 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.299382 (+ 4983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.300551 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.302561 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.304830 (+ 2269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.304965 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.321564 (+ 16599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.321570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.321578 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.321626 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.326870 (+ 5244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.327848 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.327856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.327862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.328045 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.328909 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.328913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.328915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.329286 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.329495 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.330134 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.330489 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":589,"threads_started":1}
I20260929 15:58:11.331723 31394 ts_manager.cc:194] Registered new tserver with Master: 495c68ea69d948d2888be1132f244819 (127.23.222.67:38233)
I20260929 15:58:11.378041 31891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.353993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.23.222.66:38077)
0929 15:58:11.367283 (+ 13290us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.367296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.367342 (+ 46us) server_negotiation.cc:407] Connection header received
0929 15:58:11.367498 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.367513 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.367924 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.368252 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:11.369566 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.371773 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.374809 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.376697 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.376840 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.377023 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.377512 (+ 489us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.377724 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13032,"thread_start_us":12867,"threads_started":1}
I20260929 15:58:11.378229 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.353774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:38077 (local address 127.0.0.1:38328)
0929 15:58:11.365443 (+ 11669us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:11.365482 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:11.365709 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:11.368587 (+ 2878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:11.368597 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:11.368645 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:11.369281 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.369301 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.371963 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.371968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.374646 (+ 2678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.374661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.377164 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.377171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.377318 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.377343 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 15:58:11.377461 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 15:58:11.377775 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11436,"mutex_wait_us":127,"thread_start_us":18422,"threads_started":1}
I20260929 15:58:11.382812 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.354947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.23.222.65:33817)
0929 15:58:11.366692 (+ 11745us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.366707 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.372687 (+ 5980us) server_negotiation.cc:407] Connection header received
0929 15:58:11.373088 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.373103 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.373561 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.373822 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:11.375130 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.376641 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.380129 (+ 3488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.381744 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.381885 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.382053 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.382409 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.382572 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11480,"thread_start_us":11314,"threads_started":1}
I20260929 15:58:11.382890 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.354584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:36564)
0929 15:58:11.372546 (+ 17962us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:11.372568 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:11.372831 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:11.374237 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:11.374246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:11.374289 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:11.374858 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.374876 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.376847 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.376858 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.379908 (+ 3050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.379925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.382006 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.382012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.382194 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.382227 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:58:11.382332 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 15:58:11.382468 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17800,"mutex_wait_us":30,"thread_start_us":11284,"threads_started":1}
I20260929 15:58:11.383948 31653 tablet_service.cc:1467] Processing CreateTablet for tablet a04fff9268c84de787af93f8b5b1e9cf (DEFAULT_TABLE table=test-table [id=1efb624eb0f44adcb3b98746d622fdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:11.385351 31653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04fff9268c84de787af93f8b5b1e9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:11.386567 31511 tablet_service.cc:1467] Processing CreateTablet for tablet a04fff9268c84de787af93f8b5b1e9cf (DEFAULT_TABLE table=test-table [id=1efb624eb0f44adcb3b98746d622fdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:11.388021 31511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04fff9268c84de787af93f8b5b1e9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:11.388397 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.373119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.23.222.67:38233)
0929 15:58:11.373792 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.373801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.378664 (+ 4863us) server_negotiation.cc:407] Connection header received
0929 15:58:11.378939 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.378951 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.379173 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.379417 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:11.379819 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.381131 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.385338 (+ 4207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.386846 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.386983 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.387277 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.387876 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.388053 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":332,"threads_started":1}
I20260929 15:58:11.388895 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.372823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:38233 (local address 127.0.0.1:44940)
0929 15:58:11.378463 (+ 5640us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:11.378485 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:11.378755 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:11.379397 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:11.379401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:11.379428 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:11.379619 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.379631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.383325 (+ 3694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.383330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.385131 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.385153 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.387472 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.387478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.387693 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.387728 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:58:11.387831 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 15:58:11.388429 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5533,"thread_start_us":6556,"threads_started":1}
I20260929 15:58:11.392797 31794 tablet_service.cc:1467] Processing CreateTablet for tablet a04fff9268c84de787af93f8b5b1e9cf (DEFAULT_TABLE table=test-table [id=1efb624eb0f44adcb3b98746d622fdb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:11.394225 31794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04fff9268c84de787af93f8b5b1e9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:11.400831 31896 tablet_bootstrap.cc:492] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: Bootstrap starting.
I20260929 15:58:11.402555 31897 tablet_bootstrap.cc:492] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: Bootstrap starting.
I20260929 15:58:11.405366 31896 tablet_bootstrap.cc:654] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:11.406342 31897 tablet_bootstrap.cc:654] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:11.406708 31896 log.cc:824] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:11.407600 31897 log.cc:824] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:11.409044 31900 tablet_bootstrap.cc:492] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: Bootstrap starting.
I20260929 15:58:11.410423 31896 tablet_bootstrap.cc:492] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: No bootstrap required, opened a new log
I20260929 15:58:11.410774 31896 ts_tablet_manager.cc:1397] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260929 15:58:11.412297 31900 tablet_bootstrap.cc:654] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:11.413398 31900 log.cc:824] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:11.416795 31900 tablet_bootstrap.cc:492] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: No bootstrap required, opened a new log
I20260929 15:58:11.417119 31900 ts_tablet_manager.cc:1397] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 15:58:11.418663 31897 tablet_bootstrap.cc:492] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: No bootstrap required, opened a new log
I20260929 15:58:11.418941 31897 ts_tablet_manager.cc:1397] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20260929 15:58:11.422541 31896 raft_consensus.cc:348] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:11.423036 31896 raft_consensus.cc:374] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:11.423142 31896 raft_consensus.cc:729] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83aec33916c44295876e27cc9e18e527, State: Initialized, Role: FOLLOWER
I20260929 15:58:11.423842 31896 consensus_queue.cc:260] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:11.424891 31900 raft_consensus.cc:348] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:11.425307 31900 raft_consensus.cc:374] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:11.425426 31900 raft_consensus.cc:729] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495c68ea69d948d2888be1132f244819, State: Initialized, Role: FOLLOWER
I20260929 15:58:11.426141 31900 consensus_queue.cc:260] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:11.426712 31896 ts_tablet_manager.cc:1428] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260929 15:58:11.428933 31900 ts_tablet_manager.cc:1428] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260929 15:58:11.429981 31897 raft_consensus.cc:348] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:11.430457 31897 raft_consensus.cc:374] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:11.430552 31897 raft_consensus.cc:729] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f02f25c6ca4061902c1fb1a305d7c6, State: Initialized, Role: FOLLOWER
I20260929 15:58:11.431416 31860 heartbeater.cc:503] Master 127.23.222.126:42641 was elected leader, sending a full tablet report...
I20260929 15:58:11.432487 31897 consensus_queue.cc:260] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:11.438789 31719 heartbeater.cc:503] Master 127.23.222.126:42641 was elected leader, sending a full tablet report...
I20260929 15:58:11.439051 31897 ts_tablet_manager.cc:1428] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: Time spent starting tablet: real 0.020s user 0.009s sys 0.009s
W20260929 15:58:11.500430 31581 tablet.cc:2369] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:58:11.506531 31862 tablet.cc:2369] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:11.567777 31903 raft_consensus.cc:482] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:11.568111 31903 raft_consensus.cc:504] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:11.570043 31903 leader_election.cc:288] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077), 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817)
I20260929 15:58:11.578392 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.570637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:46059 (local address 127.23.222.65:33817)
0929 15:58:11.570866 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.570870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.571030 (+ 160us) server_negotiation.cc:407] Connection header received
0929 15:58:11.571228 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.571264 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.571334 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.571434 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:11.572047 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.573156 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.576033 (+ 2877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.577344 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.577507 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.577590 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.578012 (+ 422us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.578144 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:58:11.578531 31876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.570575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.23.222.67:46059)
0929 15:58:11.570888 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:11.570910 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:11.571076 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:11.571473 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:11.571479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:11.571515 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:11.571899 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.571909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.573332 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.573339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.575855 (+ 2516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.575872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.577543 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.577548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.577768 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.577826 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 15:58:11.577985 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 15:58:11.578136 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":224,"spinlock_wait_cycles":15616}
I20260929 15:58:11.578557 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.570486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:38077 (local address 127.23.222.67:60623)
0929 15:58:11.570888 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:11.570910 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:11.571076 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:11.571473 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:11.571479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:11.571519 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:11.571881 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.571895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.573332 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.573339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.575858 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:11.575872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.577775 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.577781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:11.578031 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.578062 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 15:58:11.578140 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 15:58:11.578254 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":316,"spinlock_wait_cycles":14080}
I20260929 15:58:11.579033 31891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.570426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:60623 (local address 127.23.222.66:38077)
0929 15:58:11.570778 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.570785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.571024 (+ 239us) server_negotiation.cc:407] Connection header received
0929 15:58:11.571228 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.571254 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.571334 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.571433 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:11.572047 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.573145 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.576349 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.577621 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.577765 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.577814 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.578193 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.578326 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260929 15:58:11.579692 31531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04fff9268c84de787af93f8b5b1e9cf" candidate_uuid: "495c68ea69d948d2888be1132f244819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aec33916c44295876e27cc9e18e527" is_pre_election: true
I20260929 15:58:11.579821 31673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04fff9268c84de787af93f8b5b1e9cf" candidate_uuid: "495c68ea69d948d2888be1132f244819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" is_pre_election: true
I20260929 15:58:11.580220 31531 raft_consensus.cc:2454] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495c68ea69d948d2888be1132f244819 in term 0.
I20260929 15:58:11.580310 31673 raft_consensus.cc:2454] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495c68ea69d948d2888be1132f244819 in term 0.
I20260929 15:58:11.581025 31750 leader_election.cc:302] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 495c68ea69d948d2888be1132f244819, 83aec33916c44295876e27cc9e18e527, f3f02f25c6ca4061902c1fb1a305d7c6; no voters:
I20260929 15:58:11.581378 31903 raft_consensus.cc:2781] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:11.581545 31903 raft_consensus.cc:482] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:11.581655 31903 raft_consensus.cc:3037] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:11.583913 31903 raft_consensus.cc:504] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:11.584501 31903 leader_election.cc:288] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [CANDIDATE]: Term 1 election: Requested vote from peers f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077), 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817)
I20260929 15:58:11.585016 31673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04fff9268c84de787af93f8b5b1e9cf" candidate_uuid: "495c68ea69d948d2888be1132f244819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6"
I20260929 15:58:11.585019 31531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04fff9268c84de787af93f8b5b1e9cf" candidate_uuid: "495c68ea69d948d2888be1132f244819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aec33916c44295876e27cc9e18e527"
I20260929 15:58:11.585223 31531 raft_consensus.cc:3037] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:11.585223 31673 raft_consensus.cc:3037] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:11.588451 31673 raft_consensus.cc:2454] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495c68ea69d948d2888be1132f244819 in term 1.
I20260929 15:58:11.588451 31531 raft_consensus.cc:2454] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495c68ea69d948d2888be1132f244819 in term 1.
I20260929 15:58:11.589017 31749 leader_election.cc:302] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 495c68ea69d948d2888be1132f244819, 83aec33916c44295876e27cc9e18e527, f3f02f25c6ca4061902c1fb1a305d7c6; no voters:
I20260929 15:58:11.589455 31903 raft_consensus.cc:2781] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:11.589922 31903 raft_consensus.cc:686] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 1 LEADER]: Becoming Leader. State: Replica: 495c68ea69d948d2888be1132f244819, State: Running, Role: LEADER
I20260929 15:58:11.590451 31903 consensus_queue.cc:237] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:11.595750 30594 catalog_manager.cc:5495] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 reported cstate change: term changed from 0 to 1, leader changed from <none> to 495c68ea69d948d2888be1132f244819 (127.23.222.67). New cstate: current_term: 1 leader_uuid: "495c68ea69d948d2888be1132f244819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } health_report { overall_health: HEALTHY } } }
W20260929 15:58:11.625077 31721 tablet.cc:2369] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:11.636236 31895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.630411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.23.222.67:38233)
0929 15:58:11.630610 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.630616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.630704 (+ 88us) server_negotiation.cc:407] Connection header received
0929 15:58:11.630937 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.630942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.631037 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.631186 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:11.631648 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.632729 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.634118 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.634525 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.634622 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.635116 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:11.635686 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:11.635802 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.635967 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.636038 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 15:58:11.644850 31531 raft_consensus.cc:1261] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [term 1 FOLLOWER]: Refusing update from remote peer 495c68ea69d948d2888be1132f244819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:11.644882 31673 raft_consensus.cc:1261] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [term 1 FOLLOWER]: Refusing update from remote peer 495c68ea69d948d2888be1132f244819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:11.646011 31903 consensus_queue.cc:1035] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:11.646329 31905 consensus_queue.cc:1035] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:11.660771 31908 mvcc.cc:205] Tried to move back new op lower bound from 7334696925764775936 to 7334696925558386688. Current Snapshot: MvccSnapshot[applied={T|T < 7334696925764775936}]
I20260929 15:58:11.663761 31910 mvcc.cc:205] Tried to move back new op lower bound from 7334696925764775936 to 7334696925558386688. Current Snapshot: MvccSnapshot[applied={T|T < 7334696925764775936}]
I20260929 15:58:11.668939 31909 mvcc.cc:205] Tried to move back new op lower bound from 7334696925764775936 to 7334696925558386688. Current Snapshot: MvccSnapshot[applied={T|T < 7334696925764775936 or (T in {7334696925764775936})}]
I20260929 15:58:11.683826 30595 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:11.684130 30595 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:11.692296 30595 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78d82bc0a64f455b9e4c9bacf9b78ae0: Sending DeleteTablet for 3 replicas of tablet a04fff9268c84de787af93f8b5b1e9cf
I20260929 15:58:11.693542 31511 tablet_service.cc:1514] Processing DeleteTablet for tablet a04fff9268c84de787af93f8b5b1e9cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:11 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:36564
I20260929 15:58:11.693538 31653 tablet_service.cc:1514] Processing DeleteTablet for tablet a04fff9268c84de787af93f8b5b1e9cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:11 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:38328
I20260929 15:58:11.694520 31919 tablet_replica.cc:330] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: stopping tablet replica
I20260929 15:58:11.694495 31794 tablet_service.cc:1514] Processing DeleteTablet for tablet a04fff9268c84de787af93f8b5b1e9cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:11 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:44940
I20260929 15:58:11.695122 31919 raft_consensus.cc:2229] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:11.702149 31918 tablet_replica.cc:330] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: stopping tablet replica
I20260929 15:58:11.703616 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 30563
I20260929 15:58:11.703657 31918 raft_consensus.cc:2229] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:11.704670 31919 raft_consensus.cc:2258] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:11.704759 30625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:11.705281 31918 raft_consensus.cc:2258] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:11.707614 31918 ts_tablet_manager.cc:1905] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:11.714839 31919 ts_tablet_manager.cc:1905] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:11.715957 31918 ts_tablet_manager.cc:1918] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:11.716246 31918 log.cc:1196] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/a04fff9268c84de787af93f8b5b1e9cf
I20260929 15:58:11.717010 31918 ts_tablet_manager.cc:1939] T a04fff9268c84de787af93f8b5b1e9cf P 83aec33916c44295876e27cc9e18e527: Deleting consensus metadata
I20260929 15:58:11.725805 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.696048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.23.222.125:43047)
0929 15:58:11.696248 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.696255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.702819 (+ 6564us) server_negotiation.cc:407] Connection header received
0929 15:58:11.703031 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.703038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.703140 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.703322 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.706673 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.708169 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.712162 (+ 3993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.713178 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.713363 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.717824 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.717832 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.717836 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.717912 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.721656 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.722693 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.722700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.722704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.722857 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.723526 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.723531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.723534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.724005 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.724340 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.725142 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.725456 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 15:58:11.725862 31920 tablet_replica.cc:330] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: stopping tablet replica
I20260929 15:58:11.726378 31920 raft_consensus.cc:2229] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:11.727131 31920 raft_consensus.cc:2258] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:11.730129 31919 ts_tablet_manager.cc:1918] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:11.730336 31919 log.cc:1196] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/a04fff9268c84de787af93f8b5b1e9cf
I20260929 15:58:11.730978 31919 ts_tablet_manager.cc:1939] T a04fff9268c84de787af93f8b5b1e9cf P f3f02f25c6ca4061902c1fb1a305d7c6: Deleting consensus metadata
I20260929 15:58:11.734383 31920 ts_tablet_manager.cc:1905] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:11.742959 31920 ts_tablet_manager.cc:1918] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:11.743161 31920 log.cc:1196] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/a04fff9268c84de787af93f8b5b1e9cf
I20260929 15:58:11.743942 31920 ts_tablet_manager.cc:1939] T a04fff9268c84de787af93f8b5b1e9cf P 495c68ea69d948d2888be1132f244819: Deleting consensus metadata
I20260929 15:58:11.747546 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.702907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.23.222.124:41413)
0929 15:58:11.703085 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.703091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.729017 (+ 25926us) server_negotiation.cc:407] Connection header received
0929 15:58:11.729284 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.729290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.729374 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.729442 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.731642 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.732686 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.734721 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.735476 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.735582 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.739712 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.739720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.739724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.739784 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.743455 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.744964 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.744970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.744973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.745129 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.745657 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.745661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.745666 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.746134 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.746389 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.746759 (+ 370us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.747003 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:58:11.885577 30582 catalog_manager.cc:4841] TS 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817): tablet a04fff9268c84de787af93f8b5b1e9cf (table test-table [id=1efb624eb0f44adcb3b98746d622fdb9]) successfully deleted
I20260929 15:58:11.885731 30581 catalog_manager.cc:4841] TS 495c68ea69d948d2888be1132f244819 (127.23.222.67:38233): tablet a04fff9268c84de787af93f8b5b1e9cf (table test-table [id=1efb624eb0f44adcb3b98746d622fdb9]) successfully deleted
W20260929 15:58:11.885957 30582 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.23.222.65:33817, user_credentials={real_user=slave}} blocked reactor thread for 166675us
I20260929 15:58:11.886732 30581 catalog_manager.cc:4841] TS f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077): tablet a04fff9268c84de787af93f8b5b1e9cf (table test-table [id=1efb624eb0f44adcb3b98746d622fdb9]) successfully deleted
I20260929 15:58:11.886883 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 30563
I20260929 15:58:11.930493 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.914760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.23.222.125:43047)
0929 15:58:11.914949 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.914955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.915063 (+ 108us) server_negotiation.cc:407] Connection header received
0929 15:58:11.915401 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.915408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.915507 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.915588 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.917530 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.918305 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.920152 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.920546 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.920684 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.924056 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.924062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.924065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.924099 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.927218 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.928025 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.928030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.928035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.928183 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.928624 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.928629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.928634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.929021 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.929291 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.929659 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.930144 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:58:11.933068 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:11.915709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.23.222.124:41413)
0929 15:58:11.915891 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:11.915897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:11.916191 (+ 294us) server_negotiation.cc:407] Connection header received
0929 15:58:11.916345 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:11.916349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:11.916439 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:11.916550 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:11.918592 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.919765 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.921616 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:11.922263 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:11.922407 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:11.925519 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:11.925526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:11.925532 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:11.925582 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:11.929223 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.930320 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.930325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.930328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.930477 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:11.930895 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:11.930902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:11.930909 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:11.931285 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:11.931523 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:11.932606 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0929 15:58:11.932776 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:58:12.678076 31929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:12.677399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.65:46299)
0929 15:58:12.677804 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:12.677908 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":206,"threads_started":1}
W20260929 15:58:12.678536 31579 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:42641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
I20260929 15:58:12.683058 31930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:12.682319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.66:52543)
0929 15:58:12.682806 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:12.682917 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":262,"threads_started":1}
W20260929 15:58:12.683449 31719 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:42641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
I20260929 15:58:12.686399 31931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:12.685596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.67:58689)
0929 15:58:12.686043 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:12.686180 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":218,"threads_started":1}
W20260929 15:58:12.686816 31860 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:42641 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
I20260929 15:58:13.192099 31932 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78d82bc0a64f455b9e4c9bacf9b78ae0)
I20260929 15:58:13.192382 31933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 78d82bc0a64f455b9e4c9bacf9b78ae0)
I20260929 15:58:13.192441 31932 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } }
I20260929 15:58:13.192655 31933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } }
I20260929 15:58:13.193960 31932 leader_election.cc:288] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78d82bc0a64f455b9e4c9bacf9b78ae0 (127.23.222.126:42641), 4c34296ad5394e1c8fa12dea09f3d9eb (127.23.222.124:41413)
I20260929 15:58:13.195230 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.194078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46768)
0929 15:58:13.194674 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.194886 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":244,"threads_started":1}
I20260929 15:58:13.196295 31933 leader_election.cc:288] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 78d82bc0a64f455b9e4c9bacf9b78ae0 (127.23.222.126:42641), ba8d5e94f773430f962bb7b2dfc4633f (127.23.222.125:43047)
W20260929 15:58:13.197077 31185 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
I20260929 15:58:13.198071 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.194043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46766)
0929 15:58:13.197545 (+ 3502us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.197737 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3363,"thread_start_us":3198,"threads_started":1}
W20260929 15:58:13.198726 31379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
I20260929 15:58:13.199769 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.199022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46784)
0929 15:58:13.199326 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.199521 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260929 15:58:13.199954 31185 leader_election.cc:334] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 78d82bc0a64f455b9e4c9bacf9b78ae0 (127.23.222.126:42641): Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
I20260929 15:58:13.201202 31941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.200150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46794)
0929 15:58:13.200697 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.200882 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":299,"threads_started":1}
W20260929 15:58:13.201371 31379 leader_election.cc:334] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 78d82bc0a64f455b9e4c9bacf9b78ae0 (127.23.222.126:42641): Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
I20260929 15:58:13.204347 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.194901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.23.222.124:41413)
0929 15:58:13.197399 (+ 2498us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:13.197408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:13.197428 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:58:13.197497 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:13.197505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:13.197593 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:13.198414 (+ 821us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:13.198647 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.200187 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.202299 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.203400 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.203527 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.203580 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:13.203969 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 15:58:13.204073 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":2368,"thread_start_us":2121,"threads_started":1}
I20260929 15:58:13.204355 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.194693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:41413 (local address 127.0.0.1:54832)
0929 15:58:13.195639 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.195663 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:13.195828 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:13.197755 (+ 1927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:13.197760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:13.197788 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:13.198134 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.198151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.200331 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.200340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.202149 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.202162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.203555 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.203562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.203751 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.203789 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 15:58:13.203922 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 15:58:13.204085 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":2021,"threads_started":1}
I20260929 15:58:13.205111 31404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" is_pre_election: true
I20260929 15:58:13.205511 31404 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba8d5e94f773430f962bb7b2dfc4633f in term 1.
I20260929 15:58:13.206043 31185 leader_election.cc:302] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c34296ad5394e1c8fa12dea09f3d9eb, ba8d5e94f773430f962bb7b2dfc4633f; no voters: 78d82bc0a64f455b9e4c9bacf9b78ae0
I20260929 15:58:13.206478 31932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 15:58:13.206632 31932 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 78d82bc0a64f455b9e4c9bacf9b78ae0)
I20260929 15:58:13.206763 31932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:58:13.207005 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.198449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:43047 (local address 127.0.0.1:36902)
0929 15:58:13.198663 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.198680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:13.199570 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:13.200743 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:13.200748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:13.200804 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:13.201059 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.201068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.202659 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.202663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.204807 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.204820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.206268 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.206274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.206449 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.206482 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 15:58:13.206577 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 15:58:13.206749 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":20}
I20260929 15:58:13.207281 31939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.197707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36902 (local address 127.23.222.125:43047)
0929 15:58:13.199731 (+ 2024us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:13.199737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:13.199774 (+ 37us) server_negotiation.cc:407] Connection header received
0929 15:58:13.199828 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:13.199858 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:13.200607 (+ 749us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:13.200728 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:13.201243 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.202499 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.204976 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.206076 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.206244 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.206326 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:13.206651 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 15:58:13.206804 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":1392,"threads_started":1}
I20260929 15:58:13.207991 31210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" is_pre_election: true
I20260929 15:58:13.209368 31932 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } }
I20260929 15:58:13.209793 31210 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c34296ad5394e1c8fa12dea09f3d9eb in current term 2: Already voted for candidate ba8d5e94f773430f962bb7b2dfc4633f in this term.
I20260929 15:58:13.210096 31932 leader_election.cc:288] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [CANDIDATE]: Term 2 election: Requested vote from peers 78d82bc0a64f455b9e4c9bacf9b78ae0 (127.23.222.126:42641), 4c34296ad5394e1c8fa12dea09f3d9eb (127.23.222.124:41413)
I20260929 15:58:13.210646 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.210244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46802)
0929 15:58:13.210395 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.210499 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 15:58:13.210769 31379 leader_election.cc:302] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c34296ad5394e1c8fa12dea09f3d9eb; no voters: 78d82bc0a64f455b9e4c9bacf9b78ae0, ba8d5e94f773430f962bb7b2dfc4633f
I20260929 15:58:13.210812 31404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb"
I20260929 15:58:13.211064 31404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:58:13.211799 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.211396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46804)
0929 15:58:13.211573 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.211674 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 15:58:13.212085 31185 leader_election.cc:334] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 78d82bc0a64f455b9e4c9bacf9b78ae0 (127.23.222.126:42641): Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
I20260929 15:58:13.213425 31404 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba8d5e94f773430f962bb7b2dfc4633f in term 2.
I20260929 15:58:13.213645 31933 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 15:58:13.213878 31185 leader_election.cc:302] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4c34296ad5394e1c8fa12dea09f3d9eb, ba8d5e94f773430f962bb7b2dfc4633f; no voters: 78d82bc0a64f455b9e4c9bacf9b78ae0
I20260929 15:58:13.214224 31932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:58:13.214664 31932 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [term 2 LEADER]: Becoming Leader. State: Replica: ba8d5e94f773430f962bb7b2dfc4633f, State: Running, Role: LEADER
I20260929 15:58:13.215067 31932 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } }
I20260929 15:58:13.216724 31942 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba8d5e94f773430f962bb7b2dfc4633f. Latest consensus state: current_term: 2 leader_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:13.217156 31942 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:13.217811 31944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:13.221508 31944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:13.222373 31944 catalog_manager.cc:1259] Loaded cluster ID: 0bb7066fd387453e8024b8fb435387d6
I20260929 15:58:13.222474 31944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:13.223712 31944 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:13.224526 31944 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Loaded TSK: 0
I20260929 15:58:13.225374 31944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:58:13.245198 31199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:13.252766 31199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:13.257256 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.256826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46820)
0929 15:58:13.256989 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.257068 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 15:58:13.257725 31404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [term 2 FOLLOWER]: Refusing update from remote peer ba8d5e94f773430f962bb7b2dfc4633f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 15:58:13.258375 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.258001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46822)
0929 15:58:13.258162 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.258254 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 15:58:13.258536 31942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 15:58:13.258882 31185 consensus_peers.cc:487] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f -> Peer 78d82bc0a64f455b9e4c9bacf9b78ae0 (127.23.222.126:42641): Couldn't send request to peer 78d82bc0a64f455b9e4c9bacf9b78ae0. Status: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:58:13.260983 31933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba8d5e94f773430f962bb7b2dfc4633f. Latest consensus state: current_term: 2 leader_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:13.261233 31933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:13.262952 31932 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:13.263316 31932 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:13.263906 31933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "78d82bc0a64f455b9e4c9bacf9b78ae0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42641 } } peers { permanent_uuid: "ba8d5e94f773430f962bb7b2dfc4633f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 43047 } attrs { promote: false } } peers { permanent_uuid: "4c34296ad5394e1c8fa12dea09f3d9eb" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 41413 } attrs { promote: false } } }
I20260929 15:58:13.264168 31933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c34296ad5394e1c8fa12dea09f3d9eb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 15:58:13.294133 31935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.271963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:51238)
0929 15:58:13.272168 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.272191 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:13.272359 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:13.288211 (+ 15852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:13.288221 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:13.288262 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:13.288541 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.288555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.290119 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.290124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.292059 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.292074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.293379 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.293385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.293619 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.293660 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 15:58:13.293725 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 15:58:13.293895 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":44}
I20260929 15:58:13.294926 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.272028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.23.222.65:33817)
0929 15:58:13.287828 (+ 15800us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:13.287834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:13.287856 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:58:13.287941 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:13.287946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:13.288027 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:13.288148 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:13.288761 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.289969 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.292213 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.293199 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.294508 (+ 1309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.294589 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:13.294629 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 15:58:13.294726 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15681,"thread_start_us":15586,"threads_started":1}
I20260929 15:58:13.295648 31511 tablet_service.cc:1467] Processing CreateTablet for tablet 436c6d1124b64b78bd49ff22ae6367bc (DEFAULT_TABLE table=test-table [id=34192c1b551349908ce1f5feadd915e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:13.296118 31511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436c6d1124b64b78bd49ff22ae6367bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:13.296839 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.272703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:38077 (local address 127.0.0.1:36756)
0929 15:58:13.273212 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.273233 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:13.273375 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:13.290160 (+ 16785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:13.290167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:13.290196 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:13.290609 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.290621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.291807 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.291813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.294374 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.294389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.296013 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.296018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.296216 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.296250 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 15:58:13.296347 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 15:58:13.296477 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":116}
I20260929 15:58:13.296861 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.272865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.23.222.66:38077)
0929 15:58:13.289829 (+ 16964us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:13.289835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:13.289853 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:58:13.289921 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:13.289926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:13.290003 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:13.290733 (+ 730us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:13.290866 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.291664 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.294566 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.295804 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.295940 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.296033 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:13.296392 (+ 359us) server_negotiation.cc:299] Negotiation successful
0929 15:58:13.296541 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16842,"thread_start_us":16724,"threads_started":1}
I20260929 15:58:13.297003 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.272996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:38233 (local address 127.0.0.1:60280)
0929 15:58:13.289298 (+ 16302us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.289314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:13.289464 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:13.289997 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:13.290002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:13.290026 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:13.290512 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.290524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.291967 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.291973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.294389 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:13.294398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.296171 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.296176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:13.296290 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.296342 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 15:58:13.296441 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 15:58:13.296563 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16187,"mutex_wait_us":93,"spinlock_wait_cycles":1280,"thread_start_us":16052,"threads_started":1}
I20260929 15:58:13.297626 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.273107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.23.222.67:38233)
0929 15:58:13.287418 (+ 14311us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:13.287428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:13.289405 (+ 1977us) server_negotiation.cc:407] Connection header received
0929 15:58:13.289608 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:13.289615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:13.289694 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:13.289787 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:13.290704 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.291822 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.294565 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.295813 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.295929 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.296022 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:13.297173 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0929 15:58:13.297347 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14112,"thread_start_us":14330,"threads_started":1}
I20260929 15:58:13.297884 31653 tablet_service.cc:1467] Processing CreateTablet for tablet 436c6d1124b64b78bd49ff22ae6367bc (DEFAULT_TABLE table=test-table [id=34192c1b551349908ce1f5feadd915e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:13.298292 31653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436c6d1124b64b78bd49ff22ae6367bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:13.298269 31794 tablet_service.cc:1467] Processing CreateTablet for tablet 436c6d1124b64b78bd49ff22ae6367bc (DEFAULT_TABLE table=test-table [id=34192c1b551349908ce1f5feadd915e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:13.298585 31794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436c6d1124b64b78bd49ff22ae6367bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:13.302961 31955 tablet_bootstrap.cc:492] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527: Bootstrap starting.
I20260929 15:58:13.304687 31957 tablet_bootstrap.cc:492] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: Bootstrap starting.
I20260929 15:58:13.305297 31955 tablet_bootstrap.cc:654] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:13.306104 31957 tablet_bootstrap.cc:654] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:13.307956 31955 tablet_bootstrap.cc:492] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527: No bootstrap required, opened a new log
I20260929 15:58:13.308139 31955 ts_tablet_manager.cc:1397] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 15:58:13.308439 31957 tablet_bootstrap.cc:492] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: No bootstrap required, opened a new log
I20260929 15:58:13.308590 31957 ts_tablet_manager.cc:1397] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 15:58:13.309209 31955 raft_consensus.cc:348] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:13.309542 31955 raft_consensus.cc:374] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:13.309660 31955 raft_consensus.cc:729] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83aec33916c44295876e27cc9e18e527, State: Initialized, Role: FOLLOWER
I20260929 15:58:13.310005 31955 consensus_queue.cc:260] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:13.309245 31957 raft_consensus.cc:348] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:13.310333 31957 raft_consensus.cc:374] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:13.310437 31957 raft_consensus.cc:729] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f02f25c6ca4061902c1fb1a305d7c6, State: Initialized, Role: FOLLOWER
I20260929 15:58:13.310721 31957 consensus_queue.cc:260] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:13.313107 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.312073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.66:49685)
0929 15:58:13.312613 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.312796 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":649,"threads_started":1}
I20260929 15:58:13.314903 31956 tablet_bootstrap.cc:492] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: Bootstrap starting.
I20260929 15:58:13.315563 31957 ts_tablet_manager.cc:1428] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260929 15:58:13.315827 31720 heartbeater.cc:503] Master 127.23.222.125:43047 was elected leader, sending a full tablet report...
I20260929 15:58:13.316852 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.316422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.66:42127)
0929 15:58:13.316585 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.316690 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:58:13.317153 31955 ts_tablet_manager.cc:1428] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527: Time spent starting tablet: real 0.009s user 0.002s sys 0.007s
I20260929 15:58:13.319119 31956 tablet_bootstrap.cc:654] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:13.319541 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.317724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.65:35003)
0929 15:58:13.318871 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.319047 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":998,"thread_start_us":856,"threads_started":1}
I20260929 15:58:13.320887 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.320481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.65:56235)
0929 15:58:13.320627 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.320715 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 15:58:13.321787 31956 tablet_bootstrap.cc:492] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: No bootstrap required, opened a new log
I20260929 15:58:13.321947 31956 ts_tablet_manager.cc:1397] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260929 15:58:13.322701 31956 raft_consensus.cc:348] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:13.322918 31956 raft_consensus.cc:374] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:13.323026 31956 raft_consensus.cc:729] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495c68ea69d948d2888be1132f244819, State: Initialized, Role: FOLLOWER
I20260929 15:58:13.323347 31580 heartbeater.cc:503] Master 127.23.222.125:43047 was elected leader, sending a full tablet report...
I20260929 15:58:13.323319 31956 consensus_queue.cc:260] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:13.324175 31956 ts_tablet_manager.cc:1428] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:58:13.326318 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.324585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.67:56427)
0929 15:58:13.325068 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.325244 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":242,"threads_started":1}
I20260929 15:58:13.326560 31861 heartbeater.cc:503] Master 127.23.222.125:43047 was elected leader, sending a full tablet report...
I20260929 15:58:13.327322 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.326964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.67:42021)
0929 15:58:13.327078 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.327171 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 15:58:13.430897 31965 raft_consensus.cc:482] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:13.431125 31965 raft_consensus.cc:504] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:13.432011 31965 leader_election.cc:288] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817), f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077)
I20260929 15:58:13.432649 31531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c6d1124b64b78bd49ff22ae6367bc" candidate_uuid: "495c68ea69d948d2888be1132f244819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aec33916c44295876e27cc9e18e527" is_pre_election: true
I20260929 15:58:13.432677 31673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c6d1124b64b78bd49ff22ae6367bc" candidate_uuid: "495c68ea69d948d2888be1132f244819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" is_pre_election: true
I20260929 15:58:13.432930 31531 raft_consensus.cc:2454] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495c68ea69d948d2888be1132f244819 in term 0.
I20260929 15:58:13.432945 31673 raft_consensus.cc:2454] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495c68ea69d948d2888be1132f244819 in term 0.
I20260929 15:58:13.433533 31750 leader_election.cc:302] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 495c68ea69d948d2888be1132f244819, 83aec33916c44295876e27cc9e18e527, f3f02f25c6ca4061902c1fb1a305d7c6; no voters:
I20260929 15:58:13.433812 31965 raft_consensus.cc:2781] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:13.433971 31965 raft_consensus.cc:482] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:13.434067 31965 raft_consensus.cc:3037] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:13.436246 31965 raft_consensus.cc:504] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:13.436868 31965 leader_election.cc:288] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [CANDIDATE]: Term 1 election: Requested vote from peers 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817), f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077)
I20260929 15:58:13.437371 31531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c6d1124b64b78bd49ff22ae6367bc" candidate_uuid: "495c68ea69d948d2888be1132f244819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aec33916c44295876e27cc9e18e527"
I20260929 15:58:13.437525 31673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436c6d1124b64b78bd49ff22ae6367bc" candidate_uuid: "495c68ea69d948d2888be1132f244819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6"
I20260929 15:58:13.437613 31531 raft_consensus.cc:3037] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:13.437745 31673 raft_consensus.cc:3037] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:13.440691 31673 raft_consensus.cc:2454] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495c68ea69d948d2888be1132f244819 in term 1.
I20260929 15:58:13.440691 31531 raft_consensus.cc:2454] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495c68ea69d948d2888be1132f244819 in term 1.
I20260929 15:58:13.441224 31750 leader_election.cc:302] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 495c68ea69d948d2888be1132f244819, 83aec33916c44295876e27cc9e18e527; no voters:
I20260929 15:58:13.441565 31965 raft_consensus.cc:2781] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:13.442030 31965 raft_consensus.cc:686] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 1 LEADER]: Becoming Leader. State: Replica: 495c68ea69d948d2888be1132f244819, State: Running, Role: LEADER
I20260929 15:58:13.442341 31965 consensus_queue.cc:237] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } }
I20260929 15:58:13.443171 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.442693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.67:51169)
0929 15:58:13.442877 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.443004 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 15:58:13.445395 31200 catalog_manager.cc:5495] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 reported cstate change: term changed from 0 to 1, leader changed from <none> to 495c68ea69d948d2888be1132f244819 (127.23.222.67). New cstate: current_term: 1 leader_uuid: "495c68ea69d948d2888be1132f244819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "495c68ea69d948d2888be1132f244819" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 38233 } health_report { overall_health: HEALTHY } } }
I20260929 15:58:13.482226 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.477439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.23.222.67:38233)
0929 15:58:13.477660 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:13.477667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:13.477797 (+ 130us) server_negotiation.cc:407] Connection header received
0929 15:58:13.478026 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:13.478036 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:13.478103 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:13.478202 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:13.478541 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.479282 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.480600 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.481028 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.481160 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.481539 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:13.481635 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:13.481731 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:13.481873 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 15:58:13.481953 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:58:13.485526 31673 raft_consensus.cc:1261] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [term 1 FOLLOWER]: Refusing update from remote peer 495c68ea69d948d2888be1132f244819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:13.485563 31531 raft_consensus.cc:1261] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [term 1 FOLLOWER]: Refusing update from remote peer 495c68ea69d948d2888be1132f244819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:13.486218 31967 consensus_queue.cc:1035] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3f02f25c6ca4061902c1fb1a305d7c6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 38077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:13.486618 31965 consensus_queue.cc:1035] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83aec33916c44295876e27cc9e18e527" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 33817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:13.489859 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.489101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.65:41031)
0929 15:58:13.489434 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.489584 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260929 15:58:13.491389 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.490791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.66:42589)
0929 15:58:13.491021 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.491171 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 15:58:13.501734 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.500985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.67:50269)
0929 15:58:13.501449 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.501549 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347}
I20260929 15:58:13.502390 31963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.501739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.65:49205)
0929 15:58:13.502165 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.502250 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305}
I20260929 15:58:13.504976 31966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.504348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.67:34757)
0929 15:58:13.504728 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.504816 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
I20260929 15:58:13.507229 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.506723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.23.222.66:45853)
0929 15:58:13.506904 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.507041 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260929 15:58:13.510895 31200 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:13.511149 31200 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:13.516134 31200 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba8d5e94f773430f962bb7b2dfc4633f: Sending DeleteTablet for 3 replicas of tablet 436c6d1124b64b78bd49ff22ae6367bc
I20260929 15:58:13.517093 31511 tablet_service.cc:1514] Processing DeleteTablet for tablet 436c6d1124b64b78bd49ff22ae6367bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:13 UTC) from {username='oryx', principal='oryx/127.23.222.125@KRBTEST.COM'} at 127.0.0.1:51238
I20260929 15:58:13.517228 31653 tablet_service.cc:1514] Processing DeleteTablet for tablet 436c6d1124b64b78bd49ff22ae6367bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:13 UTC) from {username='oryx', principal='oryx/127.23.222.125@KRBTEST.COM'} at 127.0.0.1:36756
I20260929 15:58:13.517290 31794 tablet_service.cc:1514] Processing DeleteTablet for tablet 436c6d1124b64b78bd49ff22ae6367bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:13 UTC) from {username='oryx', principal='oryx/127.23.222.125@KRBTEST.COM'} at 127.0.0.1:60280
I20260929 15:58:13.519022 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 31376
I20260929 15:58:13.530206 31980 tablet_replica.cc:330] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: stopping tablet replica
I20260929 15:58:13.530421 31980 raft_consensus.cc:2229] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:13.530824 31980 raft_consensus.cc:2258] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:13.533658 31980 ts_tablet_manager.cc:1905] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:13.537585 31979 tablet_replica.cc:330] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: stopping tablet replica
I20260929 15:58:13.537778 31979 raft_consensus.cc:2229] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:13.538055 31979 raft_consensus.cc:2258] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:13.540292 31979 ts_tablet_manager.cc:1905] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:13.544921 31978 tablet_replica.cc:330] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527: stopping tablet replica
I20260929 15:58:13.545131 31978 raft_consensus.cc:2229] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:13.545423 31978 raft_consensus.cc:2258] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:13.547677 31979 ts_tablet_manager.cc:1918] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:13.547808 31979 log.cc:1196] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/436c6d1124b64b78bd49ff22ae6367bc
I20260929 15:58:13.547950 31978 ts_tablet_manager.cc:1905] T 436c6d1124b64b78bd49ff22ae6367bc P 83aec33916c44295876e27cc9e18e527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:13.548260 31979 ts_tablet_manager.cc:1939] T 436c6d1124b64b78bd49ff22ae6367bc P f3f02f25c6ca4061902c1fb1a305d7c6: Deleting consensus metadata
I20260929 15:58:13.549389 31186 catalog_manager.cc:4841] TS f3f02f25c6ca4061902c1fb1a305d7c6 (127.23.222.66:38077): tablet 436c6d1124b64b78bd49ff22ae6367bc (table test-table [id=34192c1b551349908ce1f5feadd915e7]) successfully deleted
I20260929 15:58:13.551132 31980 ts_tablet_manager.cc:1918] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:13.551291 31980 log.cc:1196] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/436c6d1124b64b78bd49ff22ae6367bc
I20260929 15:58:13.551803 31980 ts_tablet_manager.cc:1939] T 436c6d1124b64b78bd49ff22ae6367bc P 495c68ea69d948d2888be1132f244819: Deleting consensus metadata
I20260929 15:58:13.552958 31186 catalog_manager.cc:4841] TS 495c68ea69d948d2888be1132f244819 (127.23.222.67:38233): tablet 436c6d1124b64b78bd49ff22ae6367bc (table test-table [id=34192c1b551349908ce1f5feadd915e7]) successfully deleted
I20260929 15:58:13.554206 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.520085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.23.222.65:33817)
0929 15:58:13.520341 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:13.520350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:13.531081 (+ 10731us) server_negotiation.cc:407] Connection header received
0929 15:58:13.531714 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:13.531720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:13.531815 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:13.531923 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:13.534735 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.535877 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.538649 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.539387 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.539508 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.546244 (+ 6736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:13.546267 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:13.546290 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:13.546337 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:13.550581 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:13.551538 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:13.551549 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:13.551555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:13.551779 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:13.552304 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:13.552309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:13.552313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:13.552816 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:13.553098 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:13.553623 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 15:58:13.553802 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 15:58:13.554908 31573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:13.649875 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 31376
I20260929 15:58:13.670810 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 31590
I20260929 15:58:13.671445 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.670857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:51260)
0929 15:58:13.671035 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.671156 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 15:58:13.671743 31187 catalog_manager.cc:4584] TS 83aec33916c44295876e27cc9e18e527 (127.23.222.65:33817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 436c6d1124b64b78bd49ff22ae6367bc: Network error: Client connection negotiation failed: client connection to 127.23.222.65:33817: connect: Connection refused (error 111)
I20260929 15:58:13.684059 31952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.671625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.23.222.66:38077)
0929 15:58:13.671902 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:13.671909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:13.671936 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:58:13.672144 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:13.672148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:13.672210 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:13.672316 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:13.673632 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.674306 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.675687 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.676117 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.676253 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.678642 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:13.678656 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:13.678671 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:13.678706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:13.681316 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:13.682019 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:13.682025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:13.682028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:13.682143 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:13.682522 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:13.682529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:13.682532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:13.682884 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:13.683199 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:13.683517 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 15:58:13.683749 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 15:58:13.684700 31713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:13.732326 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.731843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:51268)
0929 15:58:13.732055 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.732173 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 15:58:13.733386 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.733046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:51274)
0929 15:58:13.733195 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.733274 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 15:58:13.779397 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 31590
I20260929 15:58:13.796865 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 31731
I20260929 15:58:13.801918 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.801431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:51286)
0929 15:58:13.801613 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.801733 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 15:58:13.802960 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.802595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:51288)
0929 15:58:13.802725 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.802787 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 15:58:13.810813 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.797705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.23.222.67:38233)
0929 15:58:13.797886 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:13.797891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:13.797910 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:58:13.798036 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:13.798040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:13.798101 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:13.798193 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:13.799557 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.800306 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.801701 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:13.802275 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:13.802411 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:13.805104 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:13.805119 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:13.805135 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:13.805167 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:13.808168 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:13.808846 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:13.808853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:13.808856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:13.808976 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:13.809357 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:13.809364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:13.809372 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:13.809721 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:13.810043 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:13.810290 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 15:58:13.810493 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:58:13.811501 31854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:13.826057 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.825627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46824)
0929 15:58:13.825792 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.825891 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 15:58:13.827041 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.826756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42641 (local address 127.0.0.1:46826)
0929 15:58:13.826881 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.826946 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 15:58:13.904556 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.903993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:51292)
0929 15:58:13.904193 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.904309 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 15:58:13.905689 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:13.905260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:51296)
0929 15:58:13.905413 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:13.905527 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 15:58:13.935007 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 31731
I20260929 15:58:13.957062 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 31057
I20260929 15:58:13.958249 31230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:14.082125 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:14.081438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:33817 (local address 127.0.0.1:51302)
0929 15:58:14.081728 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:14.081866 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260929 15:58:14.082360 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 31057
I20260929 15:58:14.100319 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 31156
I20260929 15:58:14.101377 31424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:14.220336 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 31156
2026-09-29T15:58:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9837 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:14 dist-test-slave-cq88 krb5kdc[31989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:14 dist-test-slave-cq88 krb5kdc[31989](info): set up 2 sockets
Sep 29 15:58:14 dist-test-slave-cq88 krb5kdc[31989](info): commencing operation
krb5kdc: starting...
W20260929 15:58:15.319125 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:58:15 dist-test-slave-cq88 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.126.keytab.
Entry for principal oryx/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.126.keytab.
I20260929 15:58:15.546649 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42125
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.126.keytab
--principal=oryx/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:42125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:15.875662 32002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:15.875989 32002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:15.876163 32002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:15.883978 32002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:15.884115 32002 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 15:58:15.884189 32002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:15.884444 32002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:15.884532 32002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:15.884621 32002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:15.895068 32002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:42125
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.126.keytab
--principal=oryx/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:15.896494 32002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:15.897897 32002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:15.911365 32008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:15.911398 32007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:15.911433 32010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:15.912292 32002 server_base.cc:1034] running on GCE node
I20260929 15:58:15.913178 32002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:15.914650 32002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:15.915869 32002 hybrid_clock.cc:648] HybridClock initialized: now 1790697495915815 us; error 32 us; skew 500 ppm
I20260929 15:58:15.916195 32002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:15 dist-test-slave-cq88 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697495, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:15.919585 32002 init.cc:377] Logged in from keytab as oryx/127.23.222.126@KRBTEST.COM (short username oryx)
I20260929 15:58:15.921115 32002 webserver.cc:460] Webserver started at http://127.23.222.126:33703/ using document root <none> and password file <none>
I20260929 15:58:15.921890 32002 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:15.922019 32002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:15.922397 32002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:15.925611 32002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "f4f681fe561d433aa3ec834a455133d8"
format_stamp: "Formatted at 2026-09-29 15:58:15 on dist-test-slave-cq88"
server_key: "24e748a85c3deafa7b6c0cdff0196be7"
server_key_iv: "a9710af42d85a5fa9249392736ccac10"
server_key_version: "encryptionkey@0"
I20260929 15:58:15.926450 32002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "f4f681fe561d433aa3ec834a455133d8"
format_stamp: "Formatted at 2026-09-29 15:58:15 on dist-test-slave-cq88"
server_key: "24e748a85c3deafa7b6c0cdff0196be7"
server_key_iv: "a9710af42d85a5fa9249392736ccac10"
server_key_version: "encryptionkey@0"
I20260929 15:58:15.931860 32002 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:58:15.935474 32017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:15.936905 32002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:15.937104 32002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "f4f681fe561d433aa3ec834a455133d8"
format_stamp: "Formatted at 2026-09-29 15:58:15 on dist-test-slave-cq88"
server_key: "24e748a85c3deafa7b6c0cdff0196be7"
server_key_iv: "a9710af42d85a5fa9249392736ccac10"
server_key_version: "encryptionkey@0"
I20260929 15:58:15.937341 32002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:15.967173 32002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:15.972190 32002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:15.972482 32002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:15.994210 32002 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:42125
I20260929 15:58:15.994216 32078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:42125 every 8 connection(s)
I20260929 15:58:15.995718 32002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:15.998258 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 32002
I20260929 15:58:15.998449 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:58:15.999094 24441 external_mini_cluster.cc:1428] Setting key 0ecd62827617c0d0514626f5da3341cd
I20260929 15:58:15.999758 32079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:58:16 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:16.013108 32079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8: Bootstrap starting.
I20260929 15:58:16.016903 32079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:16.018086 32079 log.cc:824] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:16.021744 32079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8: No bootstrap required, opened a new log
I20260929 15:58:16.022680 32082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.001153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45560 (local address 127.23.222.126:42125)
0929 15:58:16.001818 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.001830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.001871 (+ 41us) server_negotiation.cc:407] Connection header received
0929 15:58:16.003640 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.003664 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.004077 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.004559 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.006073 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.007523 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.008804 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.009504 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.009709 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.014468 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.014476 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.014491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.014540 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.017824 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.018865 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.018882 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.018892 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.019142 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.019632 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.019641 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.019647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.020255 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.020482 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.021057 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.021514 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":297,"threads_started":1}
I20260929 15:58:16.029476 32079 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f681fe561d433aa3ec834a455133d8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42125 } }
I20260929 15:58:16.029870 32079 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:16.029992 32079 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f681fe561d433aa3ec834a455133d8, State: Initialized, Role: FOLLOWER
I20260929 15:58:16.030565 32079 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f681fe561d433aa3ec834a455133d8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42125 } }
I20260929 15:58:16.030805 32079 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:16.030944 32079 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:16.031155 32079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:16.033358 32079 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f681fe561d433aa3ec834a455133d8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42125 } }
I20260929 15:58:16.033923 32079 leader_election.cc:302] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4f681fe561d433aa3ec834a455133d8; no voters:
I20260929 15:58:16.034504 32079 leader_election.cc:288] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:16.034672 32084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:16.035459 32084 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [term 1 LEADER]: Becoming Leader. State: Replica: f4f681fe561d433aa3ec834a455133d8, State: Running, Role: LEADER
I20260929 15:58:16.035988 32084 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f681fe561d433aa3ec834a455133d8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42125 } }
I20260929 15:58:16.036738 32079 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:16.039503 32086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4f681fe561d433aa3ec834a455133d8. Latest consensus state: current_term: 1 leader_uuid: "f4f681fe561d433aa3ec834a455133d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f681fe561d433aa3ec834a455133d8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42125 } } }
I20260929 15:58:16.039529 32085 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4f681fe561d433aa3ec834a455133d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4f681fe561d433aa3ec834a455133d8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42125 } } }
I20260929 15:58:16.039976 32085 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:16.040577 32086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:16.042017 32092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:16.048354 32092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:16.058229 32092 catalog_manager.cc:1347] Generated new cluster ID: b0ef102ef9034ca08c59e3fd77d973b8
I20260929 15:58:16.058369 32092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:16.067906 32092 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:16.069263 32092 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:16.077538 32092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8: Generated new TSK 0
I20260929 15:58:16.078392 32092 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab.
Entry for principal oryx/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab.
I20260929 15:58:16.140151 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42125
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab
--principal=oryx/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:16.483704 32105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:16.483995 32105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:16.484151 32105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:16.491559 32105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:16.491675 32105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:16.491861 32105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:58:16.491991 32105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:58:16.501899 32105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab
--principal=oryx/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.222.126:42125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:16.503634 32105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:16.505023 32105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:16.517930 32110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:16.517983 32113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:16.518025 32111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:16.518798 32105 server_base.cc:1034] running on GCE node
I20260929 15:58:16.519419 32105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:16.520278 32105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:16.521540 32105 hybrid_clock.cc:648] HybridClock initialized: now 1790697496521457 us; error 39 us; skew 500 ppm
I20260929 15:58:16.521983 32105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:16 dist-test-slave-cq88 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:16.525293 32105 init.cc:377] Logged in from keytab as oryx/127.23.222.65@KRBTEST.COM (short username oryx)
I20260929 15:58:16.526937 32105 webserver.cc:460] Webserver started at http://127.23.222.65:38591/ using document root <none> and password file <none>
I20260929 15:58:16.527730 32105 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:16.527822 32105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:16.528101 32105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:16.531049 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "d1508a7e9e7e45beb71585854e4cd621"
format_stamp: "Formatted at 2026-09-29 15:58:16 on dist-test-slave-cq88"
server_key: "6d15bfa355e8f701419fab3b247f3602"
server_key_iv: "b50b3141a9ba81232731113efcc32691"
server_key_version: "encryptionkey@0"
I20260929 15:58:16.531939 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "d1508a7e9e7e45beb71585854e4cd621"
format_stamp: "Formatted at 2026-09-29 15:58:16 on dist-test-slave-cq88"
server_key: "6d15bfa355e8f701419fab3b247f3602"
server_key_iv: "b50b3141a9ba81232731113efcc32691"
server_key_version: "encryptionkey@0"
I20260929 15:58:16.537508 32105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:58:16.541841 32120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:16.543524 32105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:58:16.543663 32105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "d1508a7e9e7e45beb71585854e4cd621"
format_stamp: "Formatted at 2026-09-29 15:58:16 on dist-test-slave-cq88"
server_key: "6d15bfa355e8f701419fab3b247f3602"
server_key_iv: "b50b3141a9ba81232731113efcc32691"
server_key_version: "encryptionkey@0"
I20260929 15:58:16.543919 32105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:16.555814 32105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:16.559916 32105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:16.560212 32105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:16.563076 32105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:16.563205 32105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:16.563395 32105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:16.563468 32105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:58:16 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697496, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:16.586998 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.563544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:37273 (local address 127.23.222.126:42125)
0929 15:58:16.564465 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.564470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.565340 (+ 870us) server_negotiation.cc:407] Connection header received
0929 15:58:16.567414 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.567420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.567510 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.567643 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.569978 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.571059 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.573819 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.574263 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.574408 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.580102 (+ 5694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.580111 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.580119 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.580172 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.583122 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.584323 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.584330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.584333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.584526 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.585123 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.585129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.585132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.585436 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.585601 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.586380 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.586630 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":696,"threads_started":1}
I20260929 15:58:16.588096 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.564300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42125 (local address 127.23.222.65:37273)
0929 15:58:16.565152 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:16.565200 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:16.567064 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:16.567890 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:16.567898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:16.568724 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:16.569751 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:16.569788 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.571222 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.571229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:16.573635 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:16.573652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.574552 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.574561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:16.574873 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.575792 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:16.575839 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:16.579800 (+ 3961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:16.583332 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:16.583348 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:16.583366 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:16.584130 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:16.584693 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:16.584698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:16.584703 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:16.584986 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:16.585575 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:16.585583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:16.586013 (+ 430us) client_negotiation.cc:769] Sending connection context
0929 15:58:16.586284 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 15:58:16.586842 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":395,"threads_started":1}
I20260929 15:58:16.610296 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.594231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:33449 (local address 127.23.222.126:42125)
0929 15:58:16.594482 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:16.594491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:16.595059 (+ 568us) server_negotiation.cc:407] Connection header received
0929 15:58:16.595335 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:16.595339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:16.595428 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:16.595509 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:16.597011 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.597697 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.600268 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.600693 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.600794 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.603712 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:16.603721 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:16.603728 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:16.603776 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:16.607178 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.608240 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.608248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.608251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.608417 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:16.608815 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:16.608820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:16.608824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:16.609114 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:16.609288 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:16.609702 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 15:58:16.609943 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 15:58:16.610416 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:16.594187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42125 (local address 127.23.222.65:33449)
0929 15:58:16.594927 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:16.594955 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:16.595165 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:16.595579 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:16.595587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:16.596363 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:16.596836 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:16.596852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.597836 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.597844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:16.600078 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:16.600106 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:16.600908 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:16.600918 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:16.601187 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:16.602008 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:16.602042 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:16.603479 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:16.607362 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:16.607367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:16.607375 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:16.608044 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:16.608544 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:16.608549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:16.608553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:16.608693 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:16.609240 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:16.609245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:16.609445 (+ 200us) client_negotiation.cc:769] Sending connection context
0929 15:58:16.609595 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 15:58:16.609957 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":370,"threads_started":1}
I20260929 15:58:16.622232 32105 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:35409
I20260929 15:58:16.622262 32240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:35409 every 8 connection(s)
I20260929 15:58:16.623624 32105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:16.625909 32241 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42125
I20260929 15:58:16.626140 32241 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:16.626160 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 32105
I20260929 15:58:16.626360 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:16.626837 32241 heartbeater.cc:511] Master 127.23.222.126:42125 requested a full tablet report, sending...
I20260929 15:58:16.627051 24441 external_mini_cluster.cc:1428] Setting key 473f95897fc2dd2b6bb581110e551c28
I20260929 15:58:16.629218 32034 ts_manager.cc:194] Registered new tserver with Master: d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409)
I20260929 15:58:16.630700 32034 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:37273
WARNING: no policy specified for oryx/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab.
Entry for principal oryx/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab.
I20260929 15:58:16.675094 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42125
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab
--principal=oryx/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:17.005569 32245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:17.005839 32245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:17.005981 32245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:17.013222 32245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:17.013336 32245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:17.013514 32245 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:58:17.013609 32245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:58:17.023479 32245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab
--principal=oryx/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.222.126:42125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:17.025128 32245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:17.026562 32245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:17.040063 32250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:17.040122 32251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:17.040088 32253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:17.041159 32245 server_base.cc:1034] running on GCE node
I20260929 15:58:17.041868 32245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:17.043066 32245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:17.044340 32245 hybrid_clock.cc:648] HybridClock initialized: now 1790697497044278 us; error 72 us; skew 500 ppm
I20260929 15:58:17.044673 32245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:17 dist-test-slave-cq88 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697497, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:17.048354 32245 init.cc:377] Logged in from keytab as oryx/127.23.222.66@KRBTEST.COM (short username oryx)
I20260929 15:58:17.049888 32245 webserver.cc:460] Webserver started at http://127.23.222.66:33023/ using document root <none> and password file <none>
I20260929 15:58:17.050587 32245 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:17.050674 32245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:17.050945 32245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:17.054131 32245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "54e7b7f978974e91b37e9de8daac8ab6"
format_stamp: "Formatted at 2026-09-29 15:58:17 on dist-test-slave-cq88"
server_key: "b68f4bdae954896bc625511d66d16eed"
server_key_iv: "947ac85d2246bd4f81df5f85085efc3a"
server_key_version: "encryptionkey@0"
I20260929 15:58:17.054909 32245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "54e7b7f978974e91b37e9de8daac8ab6"
format_stamp: "Formatted at 2026-09-29 15:58:17 on dist-test-slave-cq88"
server_key: "b68f4bdae954896bc625511d66d16eed"
server_key_iv: "947ac85d2246bd4f81df5f85085efc3a"
server_key_version: "encryptionkey@0"
I20260929 15:58:17.060225 32245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 15:58:17.063824 32260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:17.065322 32245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:58:17.065536 32245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "54e7b7f978974e91b37e9de8daac8ab6"
format_stamp: "Formatted at 2026-09-29 15:58:17 on dist-test-slave-cq88"
server_key: "b68f4bdae954896bc625511d66d16eed"
server_key_iv: "947ac85d2246bd4f81df5f85085efc3a"
server_key_version: "encryptionkey@0"
I20260929 15:58:17.065768 32245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:17.085907 32245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:17.090466 32245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:17.090752 32245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:17.093396 32245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:17.093514 32245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:17.093645 32245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:17.093705 32245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 15:58:17 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697497, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:17.117647 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.094066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:47057 (local address 127.23.222.126:42125)
0929 15:58:17.094261 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.094270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.095804 (+ 1534us) server_negotiation.cc:407] Connection header received
0929 15:58:17.098065 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.098073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.098157 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.098276 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:17.100448 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.101755 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.104331 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.104796 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.104935 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.109983 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:17.109991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:17.109997 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:17.110043 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:17.113595 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.114665 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.114671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.114676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.114824 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.115552 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.115559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.115564 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.115974 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:17.116138 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.117070 (+ 932us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.117295 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 15:58:17.119395 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.094792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42125 (local address 127.23.222.66:47057)
0929 15:58:17.095646 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:17.095683 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:17.097732 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:17.098512 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:17.098520 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:17.099342 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:17.100218 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.100251 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.101935 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.101942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.104169 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.104180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.105051 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.105060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.105283 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.106078 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:17.106133 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:17.109700 (+ 3567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:17.113745 (+ 4045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:17.113755 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:17.113775 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:17.114441 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:17.115005 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:17.115010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:17.115013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:17.115368 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:17.116122 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:17.116130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:17.116622 (+ 492us) client_negotiation.cc:769] Sending connection context
0929 15:58:17.116978 (+ 356us) client_negotiation.cc:240] Negotiation successful
0929 15:58:17.117591 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":416,"threads_started":1}
I20260929 15:58:17.139850 32245 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:39629
I20260929 15:58:17.139882 32379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:39629 every 8 connection(s)
I20260929 15:58:17.141566 32245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:17.143935 32380 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42125
I20260929 15:58:17.144071 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.128244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:52751 (local address 127.23.222.126:42125)
0929 15:58:17.128451 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.128457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.129493 (+ 1036us) server_negotiation.cc:407] Connection header received
0929 15:58:17.129731 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.129738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.129834 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.129979 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:17.130949 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.132015 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.133366 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.133903 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.134063 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.137038 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:17.137073 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:17.137080 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:17.137126 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:17.140924 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.141801 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.141808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.141811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.141953 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.142447 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.142454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.142459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.142943 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:17.143095 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.143531 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.143772 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 15:58:17.144181 32380 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:17.144296 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.128434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42125 (local address 127.23.222.66:52751)
0929 15:58:17.129356 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:17.129378 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:17.129573 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:17.129997 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:17.130001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:17.130516 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:17.130795 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.130804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.132170 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.132176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.133213 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.133224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.134142 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.134150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.134398 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.135282 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:17.135317 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:17.136756 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:17.141093 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:17.141098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:17.141102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:17.141660 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:17.142067 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:17.142072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:17.142076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:17.142282 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:17.143054 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:17.143059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:17.143290 (+ 231us) client_negotiation.cc:769] Sending connection context
0929 15:58:17.143422 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 15:58:17.143754 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":506,"threads_started":1}
I20260929 15:58:17.144920 32380 heartbeater.cc:511] Master 127.23.222.126:42125 requested a full tablet report, sending...
I20260929 15:58:17.146252 32034 ts_manager.cc:194] Registered new tserver with Master: 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629)
I20260929 15:58:17.147505 32034 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:47057
I20260929 15:58:17.150758 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 32245
I20260929 15:58:17.150933 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:17.151453 24441 external_mini_cluster.cc:1428] Setting key 9ca561f0c37ea341ec0f7b374cfb44c7
WARNING: no policy specified for oryx/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab.
Entry for principal oryx/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab.
I20260929 15:58:17.198868 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42125
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab
--principal=oryx/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:42651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:17.530290 32384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:17.530577 32384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:17.530733 32384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:17.538224 32384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:17.538339 32384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:17.538497 32384 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 15:58:17.538614 32384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:58:17.549206 32384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:42651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab
--principal=oryx/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.222.126:42125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:17.550522 32384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:17.551913 32384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:17.565393 32389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:17.565361 32390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:17.565425 32392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:17.566429 32384 server_base.cc:1034] running on GCE node
I20260929 15:58:17.567199 32384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:17.568300 32384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:17.569566 32384 hybrid_clock.cc:648] HybridClock initialized: now 1790697497569477 us; error 67 us; skew 500 ppm
I20260929 15:58:17.569907 32384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:17 dist-test-slave-cq88 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697497, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:17.573318 32384 init.cc:377] Logged in from keytab as oryx/127.23.222.67@KRBTEST.COM (short username oryx)
I20260929 15:58:17.574952 32384 webserver.cc:460] Webserver started at http://127.23.222.67:34559/ using document root <none> and password file <none>
I20260929 15:58:17.575829 32384 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:17.575982 32384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:17.576332 32384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:17.579531 32384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "15b16f81004c449f994a872aae65466d"
format_stamp: "Formatted at 2026-09-29 15:58:17 on dist-test-slave-cq88"
server_key: "4a04d58607aa59e988f13f477bc4f6c8"
server_key_iv: "b7c10a77e93713ce6a614a9b65efdd47"
server_key_version: "encryptionkey@0"
I20260929 15:58:17.580394 32384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "15b16f81004c449f994a872aae65466d"
format_stamp: "Formatted at 2026-09-29 15:58:17 on dist-test-slave-cq88"
server_key: "4a04d58607aa59e988f13f477bc4f6c8"
server_key_iv: "b7c10a77e93713ce6a614a9b65efdd47"
server_key_version: "encryptionkey@0"
I20260929 15:58:17.585811 32384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:17.589592 32399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:17.591136 32384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:17.591348 32384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "15b16f81004c449f994a872aae65466d"
format_stamp: "Formatted at 2026-09-29 15:58:17 on dist-test-slave-cq88"
server_key: "4a04d58607aa59e988f13f477bc4f6c8"
server_key_iv: "b7c10a77e93713ce6a614a9b65efdd47"
server_key_version: "encryptionkey@0"
I20260929 15:58:17.591601 32384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:17.609236 32384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:17.613698 32384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:17.613991 32384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:17.617034 32384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:17.617154 32384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:17.617285 32384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:17.617348 32384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:17.633652 32241 heartbeater.cc:503] Master 127.23.222.126:42125 was elected leader, sending a full tablet report...
Sep 29 15:58:17 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697497, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:17.643811 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.617372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:45423 (local address 127.23.222.126:42125)
0929 15:58:17.617619 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.617628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.619092 (+ 1464us) server_negotiation.cc:407] Connection header received
0929 15:58:17.621621 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.621628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.621720 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.621797 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:17.624277 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.625686 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.629288 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.630026 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.630217 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.635455 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:17.635465 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:17.635472 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:17.635521 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:17.639133 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.640196 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.640201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.640205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.640437 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.641109 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.641116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.641121 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.641556 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:17.641737 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.643059 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.643409 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 15:58:17.645792 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.618089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42125 (local address 127.23.222.67:45423)
0929 15:58:17.618923 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:17.618967 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:17.621261 (+ 2294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:17.622368 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:17.622380 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:17.623213 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:17.624052 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.624077 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.625942 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.625953 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.629028 (+ 3075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.629053 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.630441 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.630451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.630776 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.631707 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:17.631731 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:17.635128 (+ 3397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:17.639307 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:17.639316 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:17.639341 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:17.640057 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:17.640569 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:17.640581 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:17.640585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:17.640946 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:17.641741 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:17.641751 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:17.642470 (+ 719us) client_negotiation.cc:769] Sending connection context
0929 15:58:17.642899 (+ 429us) client_negotiation.cc:240] Negotiation successful
0929 15:58:17.643720 (+ 821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":370,"threads_started":1}
I20260929 15:58:17.665843 32384 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:39081
I20260929 15:58:17.665894 32518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:39081 every 8 connection(s)
I20260929 15:58:17.667546 32384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:17.668264 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.652087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:34635 (local address 127.23.222.126:42125)
0929 15:58:17.652263 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.652271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.653200 (+ 929us) server_negotiation.cc:407] Connection header received
0929 15:58:17.653417 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.653429 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.653489 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.653605 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:17.655278 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.656431 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.658626 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.659057 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.659206 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.661677 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:17.661682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:17.661685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:17.661717 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:17.664951 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.665900 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.665907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.665913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.666082 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.666443 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.666448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.666453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.666826 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:17.667005 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.667582 (+ 577us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.667822 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:58:17.668417 32488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.652203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42125 (local address 127.23.222.67:34635)
0929 15:58:17.653066 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:17.653091 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:17.653277 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:17.653641 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:17.653645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:17.654474 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:17.655078 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.655097 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.656577 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.656581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.658469 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.658485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.659275 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.659282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.659467 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.660256 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:17.660282 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:17.661480 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:17.665138 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:17.665143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:17.665148 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:17.665722 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:17.666209 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:17.666212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:17.666214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:17.666333 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:17.666998 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:17.667002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:17.667215 (+ 213us) client_negotiation.cc:769] Sending connection context
0929 15:58:17.667441 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 15:58:17.667814 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":473,"threads_started":1}
I20260929 15:58:17.670660 32519 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42125
I20260929 15:58:17.670920 32519 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:17.671283 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 32384
I20260929 15:58:17.671444 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:17.671845 24441 external_mini_cluster.cc:1428] Setting key 602effac2d8073c3a2db156d51eedce2
I20260929 15:58:17.671819 32519 heartbeater.cc:511] Master 127.23.222.126:42125 requested a full tablet report, sending...
I20260929 15:58:17.673313 32034 ts_manager.cc:194] Registered new tserver with Master: 15b16f81004c449f994a872aae65466d (127.23.222.67:39081)
I20260929 15:58:17.674319 32034 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:45423
I20260929 15:58:17.674938 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 15:58:17 dist-test-slave-cq88 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697497, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 15:58:17 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:17 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:17.725221 32129 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:17.714669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.23.222.126:42125)
0929 15:58:17.714893 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.714901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.715095 (+ 194us) server_negotiation.cc:407] Connection header received
0929 15:58:17.715368 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.715372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.715441 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.715550 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:17.716906 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.717635 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.718919 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.719472 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.719623 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.724955 (+ 5332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45570: BlockingRecv error: recv got EOF from 127.0.0.1:45570 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 15:58:17 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697497, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:17.741619 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.728743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.23.222.126:42125)
0929 15:58:17.728990 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.728996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.729167 (+ 171us) server_negotiation.cc:407] Connection header received
0929 15:58:17.729315 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.729318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.729398 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.729480 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:17.730623 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.731422 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.732630 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.733062 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.733152 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.736842 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:17.736847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:17.736851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:17.736886 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:17.739448 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.740037 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.740041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.740043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.740164 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:17.740464 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:17.740467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:17.740469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:17.740793 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:17.741089 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.741228 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.741396 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 15:58:17.745736 32034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:17.753012 32034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:17.781797 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.766266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:35409 (local address 127.0.0.1:48548)
0929 15:58:17.767077 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:17.767130 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:17.767980 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:17.769195 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:17.769206 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:17.769246 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:17.769798 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.769816 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.772533 (+ 2717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.772542 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.776032 (+ 3490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.776057 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.780163 (+ 4106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.780173 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.780543 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.780631 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 15:58:17.780836 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 15:58:17.781156 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":327,"threads_started":1}
I20260929 15:58:17.783290 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.766507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.23.222.65:35409)
0929 15:58:17.767829 (+ 1322us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.767840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.767875 (+ 35us) server_negotiation.cc:407] Connection header received
0929 15:58:17.768274 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.768282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.768589 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.768831 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:17.770064 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.772304 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.776306 (+ 4002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.778531 (+ 2225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.778742 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.779060 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.782489 (+ 3429us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.782800 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":977,"threads_started":1}
I20260929 15:58:17.788545 32171 tablet_service.cc:1467] Processing CreateTablet for tablet 710386a1c4ce48d09f865c58462f138e (DEFAULT_TABLE table=test-table [id=6331280290144328ad182912c2056f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:17.790293 32171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710386a1c4ce48d09f865c58462f138e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:17.792616 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.768132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.23.222.66:39629)
0929 15:58:17.784487 (+ 16355us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.784495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.784521 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:58:17.784623 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.784630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.784872 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.785610 (+ 738us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:17.785865 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.787595 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.790082 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.791459 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.791622 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.791816 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.792137 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.792313 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16191,"thread_start_us":16573,"threads_started":1}
I20260929 15:58:17.792722 32537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.767621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:39629 (local address 127.0.0.1:60226)
0929 15:58:17.782097 (+ 14476us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:17.782117 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:17.782241 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:17.785104 (+ 2863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:17.785111 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:17.785144 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:17.785434 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.785448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.787767 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.787772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.789886 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.789901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.791655 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.791664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.791926 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.791968 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 15:58:17.792075 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 15:58:17.792252 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14362,"mutex_wait_us":86,"thread_start_us":16928,"threads_started":1}
I20260929 15:58:17.793058 32540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.767568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:39081 (local address 127.0.0.1:38158)
0929 15:58:17.779685 (+ 12117us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:17.779729 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:17.779898 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:17.785007 (+ 5109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:17.785012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:17.785044 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:17.785552 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.785563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.787890 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.787898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.789830 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.789844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.791867 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.791874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.792053 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.792088 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 15:58:17.792144 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 15:58:17.792271 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11933,"mutex_wait_us":34,"thread_start_us":11743,"threads_started":1}
I20260929 15:58:17.794229 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.768258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.23.222.67:39081)
0929 15:58:17.784181 (+ 15923us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.784194 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.784238 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:58:17.784359 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.784370 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.784695 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.785043 (+ 348us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:17.785857 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.787674 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.790043 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.791622 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.793397 (+ 1775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.793618 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.793683 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.793843 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15708,"thread_start_us":15490,"threads_started":1}
I20260929 15:58:17.796752 32310 tablet_service.cc:1467] Processing CreateTablet for tablet 710386a1c4ce48d09f865c58462f138e (DEFAULT_TABLE table=test-table [id=6331280290144328ad182912c2056f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:17.797750 32449 tablet_service.cc:1467] Processing CreateTablet for tablet 710386a1c4ce48d09f865c58462f138e (DEFAULT_TABLE table=test-table [id=6331280290144328ad182912c2056f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:17.798288 32310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710386a1c4ce48d09f865c58462f138e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:17.799153 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710386a1c4ce48d09f865c58462f138e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:17.805092 32543 tablet_bootstrap.cc:492] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: Bootstrap starting.
I20260929 15:58:17.810811 32543 tablet_bootstrap.cc:654] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:17.811786 32544 tablet_bootstrap.cc:492] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: Bootstrap starting.
I20260929 15:58:17.813278 32543 log.cc:824] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:17.815101 32544 tablet_bootstrap.cc:654] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:17.816212 32544 log.cc:824] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:17.816643 32543 tablet_bootstrap.cc:492] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: No bootstrap required, opened a new log
I20260929 15:58:17.816970 32543 ts_tablet_manager.cc:1397] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 15:58:17.821485 32545 tablet_bootstrap.cc:492] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: Bootstrap starting.
I20260929 15:58:17.824033 32544 tablet_bootstrap.cc:492] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: No bootstrap required, opened a new log
I20260929 15:58:17.824358 32544 ts_tablet_manager.cc:1397] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260929 15:58:17.825898 32545 tablet_bootstrap.cc:654] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:17.827548 32545 log.cc:824] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:17.827411 32543 raft_consensus.cc:348] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:17.827939 32543 raft_consensus.cc:374] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:17.828061 32543 raft_consensus.cc:729] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1508a7e9e7e45beb71585854e4cd621, State: Initialized, Role: FOLLOWER
I20260929 15:58:17.828861 32543 consensus_queue.cc:260] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:17.830862 32543 ts_tablet_manager.cc:1428] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 15:58:17.832018 32545 tablet_bootstrap.cc:492] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: No bootstrap required, opened a new log
I20260929 15:58:17.832340 32545 ts_tablet_manager.cc:1397] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20260929 15:58:17.835719 32544 raft_consensus.cc:348] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:17.836230 32544 raft_consensus.cc:374] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:17.836342 32544 raft_consensus.cc:729] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b16f81004c449f994a872aae65466d, State: Initialized, Role: FOLLOWER
I20260929 15:58:17.837080 32544 consensus_queue.cc:260] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:17.838409 32519 heartbeater.cc:503] Master 127.23.222.126:42125 was elected leader, sending a full tablet report...
I20260929 15:58:17.839147 32544 ts_tablet_manager.cc:1428] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 15:58:17.843413 32545 raft_consensus.cc:348] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:17.843793 32545 raft_consensus.cc:374] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:17.843873 32545 raft_consensus.cc:729] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e7b7f978974e91b37e9de8daac8ab6, State: Initialized, Role: FOLLOWER
I20260929 15:58:17.844468 32545 consensus_queue.cc:260] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:17.845456 32380 heartbeater.cc:503] Master 127.23.222.126:42125 was elected leader, sending a full tablet report...
I20260929 15:58:17.846004 32545 ts_tablet_manager.cc:1428] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260929 15:58:17.876160 32242 tablet.cc:2369] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:58:17.894034 32381 tablet.cc:2369] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:58:17.919709 32520 tablet.cc:2369] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:17.988665 32550 raft_consensus.cc:482] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:17.989023 32550 raft_consensus.cc:504] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:17.990808 32550 leader_election.cc:288] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409), 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629)
I20260929 15:58:17.997735 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.991198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:39339 (local address 127.23.222.65:35409)
0929 15:58:17.991432 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.991436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.991531 (+ 95us) server_negotiation.cc:407] Connection header received
0929 15:58:17.991768 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.991772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.991833 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.991953 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:17.992418 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.993124 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.995644 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.996883 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.997050 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.997128 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.997380 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.997502 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 15:58:17.997768 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.991206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:35409 (local address 127.23.222.67:39339)
0929 15:58:17.991425 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:17.991448 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:17.991593 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:17.991975 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:17.991979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:17.992008 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:17.992237 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.992254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.993266 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.993272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.995434 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.995448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.997039 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.997045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.997221 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.997245 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 15:58:17.997355 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 15:58:17.997488 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":2}
I20260929 15:58:17.998979 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.992009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:46639 (local address 127.23.222.66:39629)
0929 15:58:17.992227 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:17.992232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:17.992684 (+ 452us) server_negotiation.cc:407] Connection header received
0929 15:58:17.992881 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:17.992886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:17.992953 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:17.993053 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:17.993685 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.994452 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.996665 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.997831 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.997997 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.998087 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:17.998579 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 15:58:17.998723 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:58:17.999034 32552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:17.991664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:39629 (local address 127.23.222.67:46639)
0929 15:58:17.992577 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:17.992591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:17.992735 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:17.993066 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:17.993070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:17.993091 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:17.993521 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.993530 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.994602 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.994607 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.996510 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:17.996526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:17.998228 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:17.998234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:17.998393 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:17.998432 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:58:17.998526 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 15:58:17.998651 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":23,"thread_start_us":620,"threads_started":1}
I20260929 15:58:17.998970 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710386a1c4ce48d09f865c58462f138e" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1508a7e9e7e45beb71585854e4cd621" is_pre_election: true
I20260929 15:58:17.999598 32196 raft_consensus.cc:2454] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 0.
I20260929 15:58:18.000224 32330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710386a1c4ce48d09f865c58462f138e" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e7b7f978974e91b37e9de8daac8ab6" is_pre_election: true
I20260929 15:58:18.000320 32400 leader_election.cc:302] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b16f81004c449f994a872aae65466d, d1508a7e9e7e45beb71585854e4cd621; no voters:
I20260929 15:58:18.000742 32550 raft_consensus.cc:2781] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:18.000748 32330 raft_consensus.cc:2454] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 0.
I20260929 15:58:18.000902 32550 raft_consensus.cc:482] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:18.001008 32550 raft_consensus.cc:3037] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:18.004180 32550 raft_consensus.cc:504] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.004825 32550 leader_election.cc:288] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 election: Requested vote from peers d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409), 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629)
I20260929 15:58:18.005412 32330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710386a1c4ce48d09f865c58462f138e" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e7b7f978974e91b37e9de8daac8ab6"
I20260929 15:58:18.005560 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710386a1c4ce48d09f865c58462f138e" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1508a7e9e7e45beb71585854e4cd621"
I20260929 15:58:18.005637 32330 raft_consensus.cc:3037] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:18.005774 32196 raft_consensus.cc:3037] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:18.008936 32196 raft_consensus.cc:2454] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 1.
I20260929 15:58:18.008936 32330 raft_consensus.cc:2454] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 1.
I20260929 15:58:18.009341 32400 leader_election.cc:302] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b16f81004c449f994a872aae65466d, 54e7b7f978974e91b37e9de8daac8ab6; no voters:
I20260929 15:58:18.009645 32550 raft_consensus.cc:2781] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:18.010108 32550 raft_consensus.cc:686] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 1 LEADER]: Becoming Leader. State: Replica: 15b16f81004c449f994a872aae65466d, State: Running, Role: LEADER
I20260929 15:58:18.010617 32550 consensus_queue.cc:237] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.015817 32033 catalog_manager.cc:5495] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d reported cstate change: term changed from 0 to 1, leader changed from <none> to 15b16f81004c449f994a872aae65466d (127.23.222.67). New cstate: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } health_report { overall_health: HEALTHY } } }
I20260929 15:58:18.074862 32542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.069158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.23.222.67:39081)
0929 15:58:18.069396 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:18.069402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:18.069553 (+ 151us) server_negotiation.cc:407] Connection header received
0929 15:58:18.069860 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:18.069865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:18.069980 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:18.070147 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:18.070513 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.071267 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.072622 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.073079 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.073220 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.073713 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:18.074257 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:18.074416 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:18.074504 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 15:58:18.074590 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:58:18.083190 32330 raft_consensus.cc:1261] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Refusing update from remote peer 15b16f81004c449f994a872aae65466d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:18.083205 32196 raft_consensus.cc:1261] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Refusing update from remote peer 15b16f81004c449f994a872aae65466d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:18.084194 32550 consensus_queue.cc:1035] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:18.084539 32553 consensus_queue.cc:1035] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:18.095534 32556 mvcc.cc:205] Tried to move back new op lower bound from 7334696952138067968 to 7334696951855837184. Current Snapshot: MvccSnapshot[applied={T|T < 7334696952138067968}]
I20260929 15:58:18.099566 32558 mvcc.cc:205] Tried to move back new op lower bound from 7334696952138067968 to 7334696951855837184. Current Snapshot: MvccSnapshot[applied={T|T < 7334696952138067968}]
I20260929 15:58:18.101568 32557 mvcc.cc:205] Tried to move back new op lower bound from 7334696952138067968 to 7334696951855837184. Current Snapshot: MvccSnapshot[applied={T|T < 7334696952138067968}]
I20260929 15:58:18.108421 32033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:18.108762 32033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:18.112921 32033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8: Sending DeleteTablet for 3 replicas of tablet 710386a1c4ce48d09f865c58462f138e
I20260929 15:58:18.114204 32449 tablet_service.cc:1514] Processing DeleteTablet for tablet 710386a1c4ce48d09f865c58462f138e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:18 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:38158
I20260929 15:58:18.114295 32310 tablet_service.cc:1514] Processing DeleteTablet for tablet 710386a1c4ce48d09f865c58462f138e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:18 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:60226
I20260929 15:58:18.114543 32171 tablet_service.cc:1514] Processing DeleteTablet for tablet 710386a1c4ce48d09f865c58462f138e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:18 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:48548
I20260929 15:58:18.116606 32033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:18.117473 32033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:18.128448 32566 tablet_replica.cc:330] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: stopping tablet replica
I20260929 15:58:18.129340 32567 tablet_replica.cc:330] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: stopping tablet replica
I20260929 15:58:18.129930 32567 raft_consensus.cc:2229] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:18.130561 32566 raft_consensus.cc:2229] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:18.131449 32448 tablet_service.cc:1467] Processing CreateTablet for tablet bfe39be5bf2247b599c03c52851d0a9a (DEFAULT_TABLE table=test-table [id=7543c2283593482a9365641e24e99005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:18.131938 32171 tablet_service.cc:1467] Processing CreateTablet for tablet bfe39be5bf2247b599c03c52851d0a9a (DEFAULT_TABLE table=test-table [id=7543c2283593482a9365641e24e99005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:18.132159 32448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe39be5bf2247b599c03c52851d0a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:18.132432 32171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe39be5bf2247b599c03c52851d0a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:18.139962 32543 tablet_bootstrap.cc:492] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: Bootstrap starting.
I20260929 15:58:18.140328 32544 tablet_bootstrap.cc:492] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: Bootstrap starting.
I20260929 15:58:18.141943 32543 tablet_bootstrap.cc:654] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:18.142146 32544 tablet_bootstrap.cc:654] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:18.131319 32310 tablet_service.cc:1467] Processing CreateTablet for tablet bfe39be5bf2247b599c03c52851d0a9a (DEFAULT_TABLE table=test-table [id=7543c2283593482a9365641e24e99005]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:18.147449 32310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe39be5bf2247b599c03c52851d0a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:18.155841 32545 tablet_bootstrap.cc:492] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: Bootstrap starting.
I20260929 15:58:18.156848 32566 raft_consensus.cc:2258] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:18.157429 32567 raft_consensus.cc:2258] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:18.157998 32545 tablet_bootstrap.cc:654] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:18.160851 32565 tablet_replica.cc:330] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: stopping tablet replica
I20260929 15:58:18.162453 32565 raft_consensus.cc:2229] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:18.162887 32567 ts_tablet_manager.cc:1905] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:18.164006 32565 raft_consensus.cc:2258] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:18.165153 32544 tablet_bootstrap.cc:492] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: No bootstrap required, opened a new log
I20260929 15:58:18.165488 32544 ts_tablet_manager.cc:1397] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.001s
I20260929 15:58:18.166494 32544 raft_consensus.cc:348] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.166810 32544 raft_consensus.cc:374] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:18.166930 32566 ts_tablet_manager.cc:1905] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:18.166968 32544 raft_consensus.cc:729] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b16f81004c449f994a872aae65466d, State: Initialized, Role: FOLLOWER
I20260929 15:58:18.167398 32544 consensus_queue.cc:260] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.168830 32544 ts_tablet_manager.cc:1428] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 15:58:18.174043 32545 tablet_bootstrap.cc:492] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: No bootstrap required, opened a new log
I20260929 15:58:18.174216 32565 ts_tablet_manager.cc:1905] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:18.174253 32545 ts_tablet_manager.cc:1397] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20260929 15:58:18.174998 32545 raft_consensus.cc:348] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.175221 32545 raft_consensus.cc:374] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:18.175356 32545 raft_consensus.cc:729] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e7b7f978974e91b37e9de8daac8ab6, State: Initialized, Role: FOLLOWER
I20260929 15:58:18.175683 32545 consensus_queue.cc:260] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.176301 32545 ts_tablet_manager.cc:1428] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 15:58:18.179608 32567 ts_tablet_manager.cc:1918] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:18.179801 32567 log.cc:1196] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/710386a1c4ce48d09f865c58462f138e
I20260929 15:58:18.180423 32567 ts_tablet_manager.cc:1939] T 710386a1c4ce48d09f865c58462f138e P d1508a7e9e7e45beb71585854e4cd621: Deleting consensus metadata
I20260929 15:58:18.180625 32543 tablet_bootstrap.cc:492] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: No bootstrap required, opened a new log
I20260929 15:58:18.180814 32543 ts_tablet_manager.cc:1397] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.009s
I20260929 15:58:18.181865 32543 raft_consensus.cc:348] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.182137 32543 raft_consensus.cc:374] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:18.182247 32543 raft_consensus.cc:729] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1508a7e9e7e45beb71585854e4cd621, State: Initialized, Role: FOLLOWER
I20260929 15:58:18.182610 32018 catalog_manager.cc:4841] TS d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409): tablet 710386a1c4ce48d09f865c58462f138e (table test-table [id=6331280290144328ad182912c2056f96]) successfully deleted
I20260929 15:58:18.182611 32543 consensus_queue.cc:260] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.182790 32565 ts_tablet_manager.cc:1918] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:18.183784 32565 log.cc:1196] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/710386a1c4ce48d09f865c58462f138e
I20260929 15:58:18.184387 32565 ts_tablet_manager.cc:1939] T 710386a1c4ce48d09f865c58462f138e P 15b16f81004c449f994a872aae65466d: Deleting consensus metadata
I20260929 15:58:18.185853 32543 ts_tablet_manager.cc:1428] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260929 15:58:18.186440 32020 catalog_manager.cc:4841] TS 15b16f81004c449f994a872aae65466d (127.23.222.67:39081): tablet 710386a1c4ce48d09f865c58462f138e (table test-table [id=6331280290144328ad182912c2056f96]) successfully deleted
I20260929 15:58:18.190375 32566 ts_tablet_manager.cc:1918] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:18.190557 32566 log.cc:1196] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/710386a1c4ce48d09f865c58462f138e
I20260929 15:58:18.191047 32566 ts_tablet_manager.cc:1939] T 710386a1c4ce48d09f865c58462f138e P 54e7b7f978974e91b37e9de8daac8ab6: Deleting consensus metadata
I20260929 15:58:18.192718 32018 catalog_manager.cc:4841] TS 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629): tablet 710386a1c4ce48d09f865c58462f138e (table test-table [id=6331280290144328ad182912c2056f96]) successfully deleted
I20260929 15:58:18.609102 32553 raft_consensus.cc:482] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:18.609304 32553 raft_consensus.cc:504] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.609863 32553 leader_election.cc:288] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409), 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629)
I20260929 15:58:18.610526 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe39be5bf2247b599c03c52851d0a9a" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1508a7e9e7e45beb71585854e4cd621" is_pre_election: true
I20260929 15:58:18.610631 32330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe39be5bf2247b599c03c52851d0a9a" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e7b7f978974e91b37e9de8daac8ab6" is_pre_election: true
I20260929 15:58:18.610785 32196 raft_consensus.cc:2454] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 0.
I20260929 15:58:18.610869 32330 raft_consensus.cc:2454] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 0.
I20260929 15:58:18.611322 32400 leader_election.cc:302] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b16f81004c449f994a872aae65466d, d1508a7e9e7e45beb71585854e4cd621; no voters:
I20260929 15:58:18.611620 32553 raft_consensus.cc:2781] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:18.611706 32553 raft_consensus.cc:482] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:18.611822 32553 raft_consensus.cc:3037] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:18.613812 32553 raft_consensus.cc:504] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.614342 32553 leader_election.cc:288] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 election: Requested vote from peers d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409), 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629)
I20260929 15:58:18.614892 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe39be5bf2247b599c03c52851d0a9a" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1508a7e9e7e45beb71585854e4cd621"
I20260929 15:58:18.614949 32330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe39be5bf2247b599c03c52851d0a9a" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e7b7f978974e91b37e9de8daac8ab6"
I20260929 15:58:18.615120 32196 raft_consensus.cc:3037] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:18.615144 32330 raft_consensus.cc:3037] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:18.618129 32330 raft_consensus.cc:2454] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 1.
I20260929 15:58:18.618129 32196 raft_consensus.cc:2454] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 1.
I20260929 15:58:18.618618 32400 leader_election.cc:302] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b16f81004c449f994a872aae65466d, d1508a7e9e7e45beb71585854e4cd621; no voters:
I20260929 15:58:18.618965 32553 raft_consensus.cc:2781] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:18.619436 32553 raft_consensus.cc:686] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 1 LEADER]: Becoming Leader. State: Replica: 15b16f81004c449f994a872aae65466d, State: Running, Role: LEADER
I20260929 15:58:18.619710 32553 consensus_queue.cc:237] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.622431 32033 catalog_manager.cc:5495] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d reported cstate change: term changed from 0 to 1, leader changed from <none> to 15b16f81004c449f994a872aae65466d (127.23.222.67). New cstate: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } health_report { overall_health: HEALTHY } } }
Sep 29 15:58:18 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697495, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.126@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:18.744870 32579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.725990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.23.222.126:42125)
0929 15:58:18.726688 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:18.726693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:18.726744 (+ 51us) server_negotiation.cc:407] Connection header received
0929 15:58:18.726940 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:18.726943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:18.727000 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:18.727143 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:18.728306 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.729092 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.730408 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.730844 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.730954 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.738929 (+ 7975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:18.738935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:18.738947 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:18.738979 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:18.741955 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:18.742782 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:18.742789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:18.742793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:18.742913 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:18.743279 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:18.743285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:18.743288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:18.743563 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:18.743715 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:18.744120 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 15:58:18.744499 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":175,"server-negotiator.queue_time_us":592,"thread_start_us":395,"threads_started":1}
I20260929 15:58:18.744920 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.725866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42125 (local address 127.0.0.1:45596)
0929 15:58:18.726665 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:18.726685 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:18.726810 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:18.727184 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:18.727187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:18.727945 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:18.728151 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.728159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.729223 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.729227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.730265 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.730276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.730998 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.731002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.731130 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.736139 (+ 5009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:18.736161 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:18.738630 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:18.742104 (+ 3474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:18.742118 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:18.742127 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:18.742657 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:18.743023 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:18.743028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:18.743031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:18.743158 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:18.743705 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:18.743714 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:18.743874 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 15:58:18.744047 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 15:58:18.744382 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":2,"thread_start_us":453,"threads_started":1}
I20260929 15:58:18.750783 32033 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:45596:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 15:58:18.751904 32033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:18.758867 32171 tablet_service.cc:1467] Processing CreateTablet for tablet ca285cba245e40c299204a56808e5724 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8799db8b3281476fbf70af064bab7f4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:58:18.758978 32310 tablet_service.cc:1467] Processing CreateTablet for tablet ca285cba245e40c299204a56808e5724 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8799db8b3281476fbf70af064bab7f4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:58:18.759173 32449 tablet_service.cc:1467] Processing CreateTablet for tablet ca285cba245e40c299204a56808e5724 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8799db8b3281476fbf70af064bab7f4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 15:58:18.759541 32171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca285cba245e40c299204a56808e5724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:18.759596 32310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca285cba245e40c299204a56808e5724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:18.759699 32449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca285cba245e40c299204a56808e5724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:18.765694 32580 tablet_bootstrap.cc:492] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: Bootstrap starting.
I20260929 15:58:18.767050 32581 tablet_bootstrap.cc:492] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: Bootstrap starting.
I20260929 15:58:18.767504 32580 tablet_bootstrap.cc:654] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:18.768826 32581 tablet_bootstrap.cc:654] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:18.769716 32582 tablet_bootstrap.cc:492] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: Bootstrap starting.
I20260929 15:58:18.770516 32580 tablet_bootstrap.cc:492] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: No bootstrap required, opened a new log
I20260929 15:58:18.770682 32580 ts_tablet_manager.cc:1397] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 15:58:18.771420 32580 raft_consensus.cc:348] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.771641 32580 raft_consensus.cc:374] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:18.771746 32580 raft_consensus.cc:729] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e7b7f978974e91b37e9de8daac8ab6, State: Initialized, Role: FOLLOWER
I20260929 15:58:18.772029 32582 tablet_bootstrap.cc:654] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:18.771999 32580 consensus_queue.cc:260] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.773855 32580 ts_tablet_manager.cc:1428] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:18.773761 32585 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.774276 32585 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:18.777395 32581 tablet_bootstrap.cc:492] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: No bootstrap required, opened a new log
I20260929 15:58:18.777508 32582 tablet_bootstrap.cc:492] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: No bootstrap required, opened a new log
I20260929 15:58:18.777601 32581 ts_tablet_manager.cc:1397] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260929 15:58:18.777928 32582 ts_tablet_manager.cc:1397] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 15:58:18.778456 32581 raft_consensus.cc:348] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.778667 32581 raft_consensus.cc:374] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:18.778561 32582 raft_consensus.cc:348] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.778786 32581 raft_consensus.cc:729] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b16f81004c449f994a872aae65466d, State: Initialized, Role: FOLLOWER
I20260929 15:58:18.778818 32582 raft_consensus.cc:374] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:18.778939 32582 raft_consensus.cc:729] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1508a7e9e7e45beb71585854e4cd621, State: Initialized, Role: FOLLOWER
I20260929 15:58:18.779071 32581 consensus_queue.cc:260] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.779158 32582 consensus_queue.cc:260] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.779567 32553 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.779776 32581 ts_tablet_manager.cc:1428] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:58:18.779824 32553 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:18.780676 32582 ts_tablet_manager.cc:1428] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:18.780652 32587 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.781131 32587 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:18.784307 32587 raft_consensus.cc:482] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:18.784505 32587 raft_consensus.cc:504] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.785336 32553 raft_consensus.cc:482] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:18.785483 32553 raft_consensus.cc:504] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.786105 32553 leader_election.cc:288] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409), 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629)
I20260929 15:58:18.786695 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca285cba245e40c299204a56808e5724" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1508a7e9e7e45beb71585854e4cd621" is_pre_election: true
I20260929 15:58:18.786962 32196 raft_consensus.cc:2454] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 0.
I20260929 15:58:18.787014 32330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca285cba245e40c299204a56808e5724" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e7b7f978974e91b37e9de8daac8ab6" is_pre_election: true
I20260929 15:58:18.787295 32330 raft_consensus.cc:2454] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 0.
I20260929 15:58:18.787451 32400 leader_election.cc:302] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b16f81004c449f994a872aae65466d, d1508a7e9e7e45beb71585854e4cd621; no voters:
I20260929 15:58:18.787767 32553 raft_consensus.cc:2781] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:18.787927 32553 raft_consensus.cc:482] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:18.788045 32553 raft_consensus.cc:3037] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:18.791977 32553 raft_consensus.cc:504] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.792552 32587 leader_election.cc:288] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629), 15b16f81004c449f994a872aae65466d (127.23.222.67:39081)
I20260929 15:58:18.792723 32553 leader_election.cc:288] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 election: Requested vote from peers d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409), 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629)
I20260929 15:58:18.793346 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca285cba245e40c299204a56808e5724" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1508a7e9e7e45beb71585854e4cd621"
I20260929 15:58:18.793366 32330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca285cba245e40c299204a56808e5724" candidate_uuid: "15b16f81004c449f994a872aae65466d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e7b7f978974e91b37e9de8daac8ab6"
I20260929 15:58:18.793648 32196 raft_consensus.cc:3037] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:18.793680 32330 raft_consensus.cc:3037] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:18.797770 32196 raft_consensus.cc:2454] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 1.
I20260929 15:58:18.797883 32330 raft_consensus.cc:2454] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b16f81004c449f994a872aae65466d in term 1.
I20260929 15:58:18.798429 32400 leader_election.cc:302] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b16f81004c449f994a872aae65466d, 54e7b7f978974e91b37e9de8daac8ab6; no voters:
I20260929 15:58:18.798810 32553 raft_consensus.cc:2781] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:18.799046 32553 raft_consensus.cc:686] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [term 1 LEADER]: Becoming Leader. State: Replica: 15b16f81004c449f994a872aae65466d, State: Running, Role: LEADER
I20260929 15:58:18.799429 32553 consensus_queue.cc:237] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:18.802991 32571 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: TxnStatusTablet state changed. Reason: New leader 15b16f81004c449f994a872aae65466d. Latest consensus state: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.803396 32571 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:58:18.804831 32033 catalog_manager.cc:5495] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d reported cstate change: term changed from 0 to 1, leader changed from <none> to 15b16f81004c449f994a872aae65466d (127.23.222.67). New cstate: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } health_report { overall_health: HEALTHY } } }
I20260929 15:58:18.807904 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.787619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:49501 (local address 127.23.222.66:39629)
0929 15:58:18.798682 (+ 11063us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:18.798690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:18.798710 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:58:18.798790 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:18.798795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:18.798879 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:18.799001 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:18.801169 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.802259 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.805101 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.806413 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.806577 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.806656 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:18.807435 (+ 779us) server_negotiation.cc:299] Negotiation successful
0929 15:58:18.807572 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10924,"thread_start_us":11778,"threads_started":1}
I20260929 15:58:18.807934 32589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.786561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:39081 (local address 127.23.222.65:41505)
0929 15:58:18.799217 (+ 12656us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:18.799261 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:18.799414 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:18.799772 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:18.799776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:18.799802 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:18.800110 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.800123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.802394 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.802400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.804902 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.804917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.806804 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.806812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.807098 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.807279 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 15:58:18.807373 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 15:58:18.807515 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12500,"mutex_wait_us":176,"thread_start_us":12938,"threads_started":1}
I20260929 15:58:18.808671 32588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.786451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:39629 (local address 127.23.222.65:49501)
0929 15:58:18.787111 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:18.787137 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:18.787337 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:18.800591 (+ 13254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:18.800595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:18.800619 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:18.800983 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.800995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.802394 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.802400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.804901 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.804917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.806815 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.806820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.807100 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.807285 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 15:58:18.808085 (+ 800us) client_negotiation.cc:240] Negotiation successful
0929 15:58:18.808275 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":299,"thread_start_us":1778,"threads_started":1}
I20260929 15:58:18.808679 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.787865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:41505 (local address 127.23.222.67:39081)
0929 15:58:18.788384 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:18.788389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:18.799356 (+ 10967us) server_negotiation.cc:407] Connection header received
0929 15:58:18.799559 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:18.799565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:18.799642 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:18.799744 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:18.800314 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.802236 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.805105 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.806446 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.806601 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.806667 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:18.808205 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0929 15:58:18.808345 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":300,"threads_started":1}
I20260929 15:58:18.809345 32593 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 15:58:18.809633 32593 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 15:58:18.809659 32330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca285cba245e40c299204a56808e5724" candidate_uuid: "d1508a7e9e7e45beb71585854e4cd621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e7b7f978974e91b37e9de8daac8ab6" is_pre_election: true
I20260929 15:58:18.809770 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca285cba245e40c299204a56808e5724" candidate_uuid: "d1508a7e9e7e45beb71585854e4cd621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b16f81004c449f994a872aae65466d" is_pre_election: true
I20260929 15:58:18.810014 32330 raft_consensus.cc:2379] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1508a7e9e7e45beb71585854e4cd621 in current term 1: Already voted for candidate 15b16f81004c449f994a872aae65466d in this term.
I20260929 15:58:18.810845 32123 leader_election.cc:302] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1508a7e9e7e45beb71585854e4cd621; no voters: 15b16f81004c449f994a872aae65466d, 54e7b7f978974e91b37e9de8daac8ab6
I20260929 15:58:18.811622 32587 raft_consensus.cc:2726] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 15:58:18.825146 32578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.819475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:39081 (local address 127.0.0.1:38172)
0929 15:58:18.819699 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:18.819723 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:18.819849 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:18.820301 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:18.820305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:18.820316 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:58:18.820461 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.820468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.821907 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.821911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.823023 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.823035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.824057 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.824061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.824213 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.824272 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:58:18.824695 (+ 423us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:58:18.824714 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 15:58:18.824784 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 15:58:18.824913 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 15:58:18.825212 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.819574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.23.222.67:39081)
0929 15:58:18.819751 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:18.819757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:18.819779 (+ 22us) server_negotiation.cc:407] Connection header received
0929 15:58:18.820046 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:18.820052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:18.820143 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:18.820243 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:18.820623 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.821744 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.823162 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.823891 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.824023 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.824452 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:18.824582 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:18.824675 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:18.824835 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 15:58:18.824924 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 15:58:18.829847 32196 raft_consensus.cc:1261] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Refusing update from remote peer 15b16f81004c449f994a872aae65466d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:18.829942 32330 raft_consensus.cc:1261] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Refusing update from remote peer 15b16f81004c449f994a872aae65466d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:18.830607 32571 consensus_queue.cc:1035] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:18.830902 32553 consensus_queue.cc:1035] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:18.834270 32585 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: TxnStatusTablet state changed. Reason: New leader 15b16f81004c449f994a872aae65466d. Latest consensus state: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.834514 32585 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:18.838228 32571 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.838505 32571 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:58:18.841012 32585 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.841238 32585 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P 54e7b7f978974e91b37e9de8daac8ab6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:18.849038 32587 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: TxnStatusTablet state changed. Reason: New leader 15b16f81004c449f994a872aae65466d. Latest consensus state: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.849346 32587 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:18.850451 32553 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.850728 32553 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d: This TxnStatusTablet replica's current role is: LEADER
I20260929 15:58:18.859812 32587 tablet_replica.cc:439] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "15b16f81004c449f994a872aae65466d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } } }
I20260929 15:58:18.860126 32587 tablet_replica.cc:442] T ca285cba245e40c299204a56808e5724 P d1508a7e9e7e45beb71585854e4cd621: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 15:58:18.865279 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.857616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:36189 (local address 127.23.222.67:39081)
0929 15:58:18.857799 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:18.857807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:18.858590 (+ 783us) server_negotiation.cc:407] Connection header received
0929 15:58:18.858942 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:18.858947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:18.859015 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:18.859104 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:18.861027 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.862012 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.863519 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.863938 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.864086 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.864511 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:18.864607 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:18.864701 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:18.864804 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 15:58:18.864907 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:58:18.865391 32606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:18.857758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:39081 (local address 127.23.222.67:36189)
0929 15:58:18.858477 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:18.858493 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:18.858813 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:18.859112 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:18.859116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:18.859126 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:58:18.859589 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.859602 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.862142 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.862147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.863384 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:18.863396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:18.864126 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:18.864133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:18.864276 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:18.864331 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:58:18.864699 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:58:18.864709 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 15:58:18.864790 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 15:58:18.864923 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":386,"threads_started":1}
I20260929 15:58:18.874913 32195 raft_consensus.cc:1261] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Refusing update from remote peer 15b16f81004c449f994a872aae65466d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:18.874913 32330 raft_consensus.cc:1261] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Refusing update from remote peer 15b16f81004c449f994a872aae65466d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:18.875507 32553 consensus_queue.cc:1035] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:18.875768 32571 consensus_queue.cc:1035] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:18.946725 32032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:18.947026 32032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:18.950804 32032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8: Sending DeleteTablet for 3 replicas of tablet bfe39be5bf2247b599c03c52851d0a9a
I20260929 15:58:18.951785 32310 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe39be5bf2247b599c03c52851d0a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:18 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:60226
I20260929 15:58:18.951841 32448 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe39be5bf2247b599c03c52851d0a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:18 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:38158
I20260929 15:58:18.952443 32171 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe39be5bf2247b599c03c52851d0a9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:18 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:48548
I20260929 15:58:18.962095 32611 tablet_replica.cc:330] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: stopping tablet replica
I20260929 15:58:18.963614 32611 raft_consensus.cc:2229] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:18.964006 32611 raft_consensus.cc:2258] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:18.967434 32611 ts_tablet_manager.cc:1905] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:18.968219 32613 tablet_replica.cc:330] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: stopping tablet replica
I20260929 15:58:18.968557 32613 raft_consensus.cc:2229] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:18.968972 32613 raft_consensus.cc:2258] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:18.969184 32612 tablet_replica.cc:330] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: stopping tablet replica
I20260929 15:58:18.969424 32612 raft_consensus.cc:2229] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:18.969936 32612 raft_consensus.cc:2258] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:18.971099 32612 ts_tablet_manager.cc:1905] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:18.971468 32613 ts_tablet_manager.cc:1905] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:18.976423 32611 ts_tablet_manager.cc:1918] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:18.976579 32611 log.cc:1196] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/bfe39be5bf2247b599c03c52851d0a9a
I20260929 15:58:18.977125 32611 ts_tablet_manager.cc:1939] T bfe39be5bf2247b599c03c52851d0a9a P 54e7b7f978974e91b37e9de8daac8ab6: Deleting consensus metadata
I20260929 15:58:18.978385 32018 catalog_manager.cc:4841] TS 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629): tablet bfe39be5bf2247b599c03c52851d0a9a (table test-table [id=7543c2283593482a9365641e24e99005]) successfully deleted
I20260929 15:58:18.982921 32613 ts_tablet_manager.cc:1918] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:18.983153 32613 log.cc:1196] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/bfe39be5bf2247b599c03c52851d0a9a
I20260929 15:58:18.983727 32613 ts_tablet_manager.cc:1939] T bfe39be5bf2247b599c03c52851d0a9a P d1508a7e9e7e45beb71585854e4cd621: Deleting consensus metadata
I20260929 15:58:18.984117 32612 ts_tablet_manager.cc:1918] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:18.984282 32612 log.cc:1196] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/bfe39be5bf2247b599c03c52851d0a9a
I20260929 15:58:18.984782 32612 ts_tablet_manager.cc:1939] T bfe39be5bf2247b599c03c52851d0a9a P 15b16f81004c449f994a872aae65466d: Deleting consensus metadata
I20260929 15:58:18.985080 32018 catalog_manager.cc:4841] TS d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409): tablet bfe39be5bf2247b599c03c52851d0a9a (table test-table [id=7543c2283593482a9365641e24e99005]) successfully deleted
I20260929 15:58:18.985913 32020 catalog_manager.cc:4841] TS 15b16f81004c449f994a872aae65466d (127.23.222.67:39081): tablet bfe39be5bf2247b599c03c52851d0a9a (table test-table [id=7543c2283593482a9365641e24e99005]) successfully deleted
Sep 29 15:58:18 dist-test-slave-cq88 krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 15:58:19 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:19 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:19.022066 32579 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:19.011692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.23.222.126:42125)
0929 15:58:19.011937 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.011946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.012511 (+ 565us) server_negotiation.cc:407] Connection header received
0929 15:58:19.012747 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.012752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.012818 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.012931 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:19.014290 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.015059 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.016360 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.016798 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.016936 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.021861 (+ 4925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45608: BlockingRecv error: recv got EOF from 127.0.0.1:45608 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 15:58:19 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM
I20260929 15:58:19.042191 32579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.026085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.23.222.126:42125)
0929 15:58:19.026300 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.026307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.026622 (+ 315us) server_negotiation.cc:407] Connection header received
0929 15:58:19.026766 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.026772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.026864 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.026988 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:19.028388 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.029583 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.030930 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.031600 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.031738 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.035955 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:19.035963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:19.035968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:19.036017 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:19.039409 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.040086 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.040094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.040098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.040239 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.040634 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.040639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.040644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.040935 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:19.041219 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.041576 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.041810 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:58:19.046552 32032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:19.047519 32032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:19.055474 32171 tablet_service.cc:1467] Processing CreateTablet for tablet ccee868f4946456f9c0522687b012d68 (DEFAULT_TABLE table=test-table [id=bf32e3cbf8924c229dad1958c6d79b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:19.055830 32448 tablet_service.cc:1467] Processing CreateTablet for tablet ccee868f4946456f9c0522687b012d68 (DEFAULT_TABLE table=test-table [id=bf32e3cbf8924c229dad1958c6d79b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:19.055832 32310 tablet_service.cc:1467] Processing CreateTablet for tablet ccee868f4946456f9c0522687b012d68 (DEFAULT_TABLE table=test-table [id=bf32e3cbf8924c229dad1958c6d79b77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:19.055999 32171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee868f4946456f9c0522687b012d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:19.056278 32448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee868f4946456f9c0522687b012d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:19.056283 32310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee868f4946456f9c0522687b012d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:19.062915 32580 tablet_bootstrap.cc:492] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: Bootstrap starting.
I20260929 15:58:19.063499 32581 tablet_bootstrap.cc:492] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: Bootstrap starting.
I20260929 15:58:19.064749 32580 tablet_bootstrap.cc:654] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:19.065263 32581 tablet_bootstrap.cc:654] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:19.067345 32580 tablet_bootstrap.cc:492] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: No bootstrap required, opened a new log
I20260929 15:58:19.067556 32580 ts_tablet_manager.cc:1397] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 15:58:19.068544 32580 raft_consensus.cc:348] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.068985 32580 raft_consensus.cc:374] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:19.069085 32580 raft_consensus.cc:729] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e7b7f978974e91b37e9de8daac8ab6, State: Initialized, Role: FOLLOWER
I20260929 15:58:19.069895 32580 consensus_queue.cc:260] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.070815 32580 ts_tablet_manager.cc:1428] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:19.071314 32582 tablet_bootstrap.cc:492] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: Bootstrap starting.
I20260929 15:58:19.073632 32582 tablet_bootstrap.cc:654] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:19.080101 32582 tablet_bootstrap.cc:492] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: No bootstrap required, opened a new log
I20260929 15:58:19.080621 32582 ts_tablet_manager.cc:1397] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 15:58:19.081876 32582 raft_consensus.cc:348] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.082275 32582 raft_consensus.cc:374] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:19.082373 32582 raft_consensus.cc:729] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1508a7e9e7e45beb71585854e4cd621, State: Initialized, Role: FOLLOWER
I20260929 15:58:19.082718 32582 consensus_queue.cc:260] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.083730 32582 ts_tablet_manager.cc:1428] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:19.086076 32581 tablet_bootstrap.cc:492] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: No bootstrap required, opened a new log
I20260929 15:58:19.086278 32581 ts_tablet_manager.cc:1397] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.016s
I20260929 15:58:19.087131 32581 raft_consensus.cc:348] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.087478 32581 raft_consensus.cc:374] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:19.087610 32581 raft_consensus.cc:729] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b16f81004c449f994a872aae65466d, State: Initialized, Role: FOLLOWER
I20260929 15:58:19.087894 32581 consensus_queue.cc:260] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.088441 32581 ts_tablet_manager.cc:1428] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 15:58:19.302539 32587 raft_consensus.cc:482] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:19.302803 32587 raft_consensus.cc:504] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.303542 32587 leader_election.cc:288] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629), 15b16f81004c449f994a872aae65466d (127.23.222.67:39081)
I20260929 15:58:19.304091 32585 raft_consensus.cc:482] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:19.304185 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee868f4946456f9c0522687b012d68" candidate_uuid: "d1508a7e9e7e45beb71585854e4cd621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b16f81004c449f994a872aae65466d" is_pre_election: true
I20260929 15:58:19.304363 32585 raft_consensus.cc:504] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.304531 32469 raft_consensus.cc:2454] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1508a7e9e7e45beb71585854e4cd621 in term 0.
I20260929 15:58:19.304497 32329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee868f4946456f9c0522687b012d68" candidate_uuid: "d1508a7e9e7e45beb71585854e4cd621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e7b7f978974e91b37e9de8daac8ab6" is_pre_election: true
I20260929 15:58:19.305017 32123 leader_election.cc:302] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b16f81004c449f994a872aae65466d, d1508a7e9e7e45beb71585854e4cd621; no voters:
I20260929 15:58:19.305088 32329 raft_consensus.cc:2454] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1508a7e9e7e45beb71585854e4cd621 in term 0.
I20260929 15:58:19.305378 32587 raft_consensus.cc:2781] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:19.305524 32587 raft_consensus.cc:482] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:19.305639 32587 raft_consensus.cc:3037] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:19.306469 32585 leader_election.cc:288] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409), 15b16f81004c449f994a872aae65466d (127.23.222.67:39081)
I20260929 15:58:19.309258 32587 raft_consensus.cc:504] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.311220 32587 leader_election.cc:288] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [CANDIDATE]: Term 1 election: Requested vote from peers 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629), 15b16f81004c449f994a872aae65466d (127.23.222.67:39081)
I20260929 15:58:19.311172 32329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee868f4946456f9c0522687b012d68" candidate_uuid: "d1508a7e9e7e45beb71585854e4cd621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54e7b7f978974e91b37e9de8daac8ab6"
I20260929 15:58:19.311419 32329 raft_consensus.cc:3037] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:19.312003 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee868f4946456f9c0522687b012d68" candidate_uuid: "d1508a7e9e7e45beb71585854e4cd621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b16f81004c449f994a872aae65466d"
I20260929 15:58:19.312309 32469 raft_consensus.cc:3037] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:19.314786 32329 raft_consensus.cc:2454] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1508a7e9e7e45beb71585854e4cd621 in term 1.
I20260929 15:58:19.315299 32121 leader_election.cc:302] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e7b7f978974e91b37e9de8daac8ab6, d1508a7e9e7e45beb71585854e4cd621; no voters:
I20260929 15:58:19.315665 32587 raft_consensus.cc:2781] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:19.316099 32469 raft_consensus.cc:2454] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1508a7e9e7e45beb71585854e4cd621 in term 1.
I20260929 15:58:19.316753 32587 raft_consensus.cc:686] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 1 LEADER]: Becoming Leader. State: Replica: d1508a7e9e7e45beb71585854e4cd621, State: Running, Role: LEADER
I20260929 15:58:19.317499 32587 consensus_queue.cc:237] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.318199 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.306970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:58011 (local address 127.23.222.65:35409)
0929 15:58:19.310812 (+ 3842us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.310817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.310843 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:58:19.310910 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.310918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.311025 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.311651 (+ 626us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:19.311817 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.312864 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.315456 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.317028 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.317186 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.317275 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.317755 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.317894 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3730,"thread_start_us":3640,"threads_started":1}
I20260929 15:58:19.318244 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.306792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:35409 (local address 127.23.222.66:58011)
0929 15:58:19.309741 (+ 2949us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:19.309764 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:19.309907 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:19.311190 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:19.311198 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:19.311222 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:19.311496 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.311510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.313029 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.313035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.315185 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.315199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.317235 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.317244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.317481 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.317527 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 15:58:19.317632 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 15:58:19.317818 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2806,"thread_start_us":2739,"threads_started":1}
I20260929 15:58:19.319773 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee868f4946456f9c0522687b012d68" candidate_uuid: "54e7b7f978974e91b37e9de8daac8ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1508a7e9e7e45beb71585854e4cd621" is_pre_election: true
I20260929 15:58:19.321213 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.307186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:40881 (local address 127.23.222.67:39081)
0929 15:58:19.307499 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.307511 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.310763 (+ 3252us) server_negotiation.cc:407] Connection header received
0929 15:58:19.310977 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.310987 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.311086 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.311224 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:19.312603 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.313975 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.318056 (+ 4081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.319900 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.320095 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.320216 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.320888 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.321026 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":109}
I20260929 15:58:19.321435 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.306813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:39081 (local address 127.23.222.66:40881)
0929 15:58:19.310649 (+ 3836us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:19.310669 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:19.310806 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:19.311602 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:19.311609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:19.311643 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:19.312267 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.312299 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.314252 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.314258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.317761 (+ 3503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.317777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.320428 (+ 2651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.320437 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.320661 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.320706 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 15:58:19.320810 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 15:58:19.320971 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3671,"thread_start_us":3537,"threads_started":1}
I20260929 15:58:19.322072 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee868f4946456f9c0522687b012d68" candidate_uuid: "54e7b7f978974e91b37e9de8daac8ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b16f81004c449f994a872aae65466d" is_pre_election: true
I20260929 15:58:19.321909 32034 catalog_manager.cc:5495] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65). New cstate: current_term: 1 leader_uuid: "d1508a7e9e7e45beb71585854e4cd621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } health_report { overall_health: UNKNOWN } } }
I20260929 15:58:19.322402 32469 raft_consensus.cc:2379] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54e7b7f978974e91b37e9de8daac8ab6 in current term 1: Already voted for candidate d1508a7e9e7e45beb71585854e4cd621 in this term.
I20260929 15:58:19.322942 32263 leader_election.cc:302] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54e7b7f978974e91b37e9de8daac8ab6; no voters: 15b16f81004c449f994a872aae65466d, d1508a7e9e7e45beb71585854e4cd621
I20260929 15:58:19.323525 32585 raft_consensus.cc:2726] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 15:58:19.378628 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.373153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48556 (local address 127.23.222.65:35409)
0929 15:58:19.373420 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.373428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.373527 (+ 99us) server_negotiation.cc:407] Connection header received
0929 15:58:19.373830 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.373835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.373914 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.374081 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:19.374440 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.375144 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.376495 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.376901 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.377033 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.377503 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:19.378045 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:19.378213 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.378294 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.378376 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:58:19.382225 32329 raft_consensus.cc:1261] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Refusing update from remote peer d1508a7e9e7e45beb71585854e4cd621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:19.382349 32469 raft_consensus.cc:1261] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Refusing update from remote peer d1508a7e9e7e45beb71585854e4cd621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:19.382985 32632 consensus_queue.cc:1035] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:19.383327 32587 consensus_queue.cc:1035] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:19.400614 32033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:19.400835 32033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:19.404050 32033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8: Sending DeleteTablet for 3 replicas of tablet ccee868f4946456f9c0522687b012d68
I20260929 15:58:19.404908 32171 tablet_service.cc:1514] Processing DeleteTablet for tablet ccee868f4946456f9c0522687b012d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:19 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:48548
I20260929 15:58:19.405125 32310 tablet_service.cc:1514] Processing DeleteTablet for tablet ccee868f4946456f9c0522687b012d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:19 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:60226
I20260929 15:58:19.405247 32613 tablet_replica.cc:330] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: stopping tablet replica
I20260929 15:58:19.405457 32613 raft_consensus.cc:2229] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:19.405985 32613 raft_consensus.cc:2258] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:19.406306 32611 tablet_replica.cc:330] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: stopping tablet replica
I20260929 15:58:19.406518 32611 raft_consensus.cc:2229] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:19.406759 32611 raft_consensus.cc:2258] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:19.407022 32033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 15:58:19.407614 32448 tablet_service.cc:1514] Processing DeleteTablet for tablet ccee868f4946456f9c0522687b012d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:19 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:38158
I20260929 15:58:19.407888 32612 tablet_replica.cc:330] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: stopping tablet replica
W20260929 15:58:19.408022 32033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:19.408104 32612 raft_consensus.cc:2229] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:19.408361 32612 raft_consensus.cc:2258] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:19.409458 32611 ts_tablet_manager.cc:1905] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:19.409585 32613 ts_tablet_manager.cc:1905] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:19.414201 32611 ts_tablet_manager.cc:1918] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:19.414322 32611 log.cc:1196] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/ccee868f4946456f9c0522687b012d68
I20260929 15:58:19.414601 32611 ts_tablet_manager.cc:1939] T ccee868f4946456f9c0522687b012d68 P 54e7b7f978974e91b37e9de8daac8ab6: Deleting consensus metadata
I20260929 15:58:19.415683 32018 catalog_manager.cc:4841] TS 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629): tablet ccee868f4946456f9c0522687b012d68 (table test-table [id=bf32e3cbf8924c229dad1958c6d79b77]) successfully deleted
I20260929 15:58:19.418280 32612 ts_tablet_manager.cc:1905] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:19.418608 32613 ts_tablet_manager.cc:1918] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:19.418802 32613 log.cc:1196] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/ccee868f4946456f9c0522687b012d68
I20260929 15:58:19.419265 32613 ts_tablet_manager.cc:1939] T ccee868f4946456f9c0522687b012d68 P d1508a7e9e7e45beb71585854e4cd621: Deleting consensus metadata
I20260929 15:58:19.421739 32018 catalog_manager.cc:4841] TS d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409): tablet ccee868f4946456f9c0522687b012d68 (table test-table [id=bf32e3cbf8924c229dad1958c6d79b77]) successfully deleted
I20260929 15:58:19.423437 32612 ts_tablet_manager.cc:1918] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:19.423611 32612 log.cc:1196] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/ccee868f4946456f9c0522687b012d68
I20260929 15:58:19.424000 32612 ts_tablet_manager.cc:1939] T ccee868f4946456f9c0522687b012d68 P 15b16f81004c449f994a872aae65466d: Deleting consensus metadata
I20260929 15:58:19.424984 32020 catalog_manager.cc:4841] TS 15b16f81004c449f994a872aae65466d (127.23.222.67:39081): tablet ccee868f4946456f9c0522687b012d68 (table test-table [id=bf32e3cbf8924c229dad1958c6d79b77]) successfully deleted
I20260929 15:58:19.427906 32171 tablet_service.cc:1467] Processing CreateTablet for tablet a8dee00ca21f44c6a25c61e66ed16b20 (DEFAULT_TABLE table=test-table [id=5f43fa3335584f8b812440b0d746015c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:19.428040 32310 tablet_service.cc:1467] Processing CreateTablet for tablet a8dee00ca21f44c6a25c61e66ed16b20 (DEFAULT_TABLE table=test-table [id=5f43fa3335584f8b812440b0d746015c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:19.428215 32448 tablet_service.cc:1467] Processing CreateTablet for tablet a8dee00ca21f44c6a25c61e66ed16b20 (DEFAULT_TABLE table=test-table [id=5f43fa3335584f8b812440b0d746015c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:19.428408 32171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8dee00ca21f44c6a25c61e66ed16b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:19.428481 32310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8dee00ca21f44c6a25c61e66ed16b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:19.428586 32448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8dee00ca21f44c6a25c61e66ed16b20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:19.435093 32581 tablet_bootstrap.cc:492] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: Bootstrap starting.
I20260929 15:58:19.436443 32581 tablet_bootstrap.cc:654] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:19.436621 32580 tablet_bootstrap.cc:492] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: Bootstrap starting.
I20260929 15:58:19.437593 32582 tablet_bootstrap.cc:492] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: Bootstrap starting.
I20260929 15:58:19.438275 32580 tablet_bootstrap.cc:654] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:19.439085 32582 tablet_bootstrap.cc:654] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:19.441700 32582 tablet_bootstrap.cc:492] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: No bootstrap required, opened a new log
I20260929 15:58:19.441943 32582 ts_tablet_manager.cc:1397] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 15:58:19.443205 32582 raft_consensus.cc:348] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.443796 32582 raft_consensus.cc:374] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:19.443982 32582 raft_consensus.cc:729] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1508a7e9e7e45beb71585854e4cd621, State: Initialized, Role: FOLLOWER
I20260929 15:58:19.444383 32582 consensus_queue.cc:260] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.446226 32582 ts_tablet_manager.cc:1428] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 15:58:19.449817 32580 tablet_bootstrap.cc:492] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: No bootstrap required, opened a new log
I20260929 15:58:19.449817 32581 tablet_bootstrap.cc:492] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: No bootstrap required, opened a new log
I20260929 15:58:19.449985 32581 ts_tablet_manager.cc:1397] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260929 15:58:19.450104 32580 ts_tablet_manager.cc:1397] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.007s
I20260929 15:58:19.450830 32581 raft_consensus.cc:348] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.450897 32580 raft_consensus.cc:348] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.451138 32581 raft_consensus.cc:374] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:19.451136 32580 raft_consensus.cc:374] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:19.451260 32581 raft_consensus.cc:729] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b16f81004c449f994a872aae65466d, State: Initialized, Role: FOLLOWER
I20260929 15:58:19.451267 32580 raft_consensus.cc:729] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54e7b7f978974e91b37e9de8daac8ab6, State: Initialized, Role: FOLLOWER
I20260929 15:58:19.451498 32580 consensus_queue.cc:260] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.451498 32581 consensus_queue.cc:260] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.452172 32581 ts_tablet_manager.cc:1428] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 15:58:19.452164 32580 ts_tablet_manager.cc:1428] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 15:58:19.475121 32585 raft_consensus.cc:482] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:19.475432 32585 raft_consensus.cc:504] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.476128 32585 leader_election.cc:288] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409), 15b16f81004c449f994a872aae65466d (127.23.222.67:39081)
I20260929 15:58:19.476715 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8dee00ca21f44c6a25c61e66ed16b20" candidate_uuid: "54e7b7f978974e91b37e9de8daac8ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1508a7e9e7e45beb71585854e4cd621" is_pre_election: true
I20260929 15:58:19.476758 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8dee00ca21f44c6a25c61e66ed16b20" candidate_uuid: "54e7b7f978974e91b37e9de8daac8ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b16f81004c449f994a872aae65466d" is_pre_election: true
I20260929 15:58:19.476972 32196 raft_consensus.cc:2454] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e7b7f978974e91b37e9de8daac8ab6 in term 0.
I20260929 15:58:19.477072 32469 raft_consensus.cc:2454] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54e7b7f978974e91b37e9de8daac8ab6 in term 0.
I20260929 15:58:19.477488 32261 leader_election.cc:302] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54e7b7f978974e91b37e9de8daac8ab6, d1508a7e9e7e45beb71585854e4cd621; no voters:
I20260929 15:58:19.477828 32585 raft_consensus.cc:2781] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:19.477978 32585 raft_consensus.cc:482] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:19.478103 32585 raft_consensus.cc:3037] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:19.480337 32585 raft_consensus.cc:504] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.481019 32585 leader_election.cc:288] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [CANDIDATE]: Term 1 election: Requested vote from peers d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409), 15b16f81004c449f994a872aae65466d (127.23.222.67:39081)
I20260929 15:58:19.481539 32469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8dee00ca21f44c6a25c61e66ed16b20" candidate_uuid: "54e7b7f978974e91b37e9de8daac8ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15b16f81004c449f994a872aae65466d"
I20260929 15:58:19.481565 32196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8dee00ca21f44c6a25c61e66ed16b20" candidate_uuid: "54e7b7f978974e91b37e9de8daac8ab6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1508a7e9e7e45beb71585854e4cd621"
I20260929 15:58:19.481786 32469 raft_consensus.cc:3037] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:19.481799 32196 raft_consensus.cc:3037] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:19.484130 32469 raft_consensus.cc:2454] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e7b7f978974e91b37e9de8daac8ab6 in term 1.
I20260929 15:58:19.484575 32263 leader_election.cc:302] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b16f81004c449f994a872aae65466d, 54e7b7f978974e91b37e9de8daac8ab6; no voters:
I20260929 15:58:19.484874 32585 raft_consensus.cc:2781] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:19.485019 32196 raft_consensus.cc:2454] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54e7b7f978974e91b37e9de8daac8ab6 in term 1.
I20260929 15:58:19.485381 32585 raft_consensus.cc:686] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 LEADER]: Becoming Leader. State: Replica: 54e7b7f978974e91b37e9de8daac8ab6, State: Running, Role: LEADER
I20260929 15:58:19.485865 32585 consensus_queue.cc:237] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } }
I20260929 15:58:19.489686 32033 catalog_manager.cc:5495] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66). New cstate: current_term: 1 leader_uuid: "54e7b7f978974e91b37e9de8daac8ab6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54e7b7f978974e91b37e9de8daac8ab6" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 39629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 } health_report { overall_health: UNKNOWN } } }
I20260929 15:58:19.515440 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.508712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.23.222.66:39629)
0929 15:58:19.509330 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.509335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.509354 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:58:19.509441 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.509445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.509520 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.509717 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:19.510131 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.510869 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.512906 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.513342 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.513454 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.514008 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:19.514637 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:19.514788 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.514931 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.515006 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":354,"threads_started":1}
I20260929 15:58:19.532146 32591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.525730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:43597 (local address 127.23.222.67:39081)
0929 15:58:19.525928 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.525934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.526520 (+ 586us) server_negotiation.cc:407] Connection header received
0929 15:58:19.526680 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.526686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.526777 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.526876 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:19.527391 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.528513 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.529923 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.530569 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.530711 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.531218 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:19.531392 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:19.531487 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.531719 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.531812 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:58:19.532193 32648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.525720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:39081 (local address 127.23.222.66:43597)
0929 15:58:19.526421 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:19.526436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:19.526555 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:19.526899 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:19.526902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:19.526912 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:58:19.527184 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.527207 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.528668 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.528673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.529777 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.529789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.530830 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.530836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.530964 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.531023 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:58:19.531545 (+ 522us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:58:19.531559 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 15:58:19.531695 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 15:58:19.531831 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":362,"threads_started":1}
I20260929 15:58:19.541940 32196 raft_consensus.cc:1261] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Refusing update from remote peer 54e7b7f978974e91b37e9de8daac8ab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:19.541930 32469 raft_consensus.cc:1261] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Refusing update from remote peer 54e7b7f978974e91b37e9de8daac8ab6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:19.542681 32642 consensus_queue.cc:1035] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1508a7e9e7e45beb71585854e4cd621" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 35409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:19.543110 32585 consensus_queue.cc:1035] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15b16f81004c449f994a872aae65466d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:19.588019 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.580900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:47287 (local address 127.23.222.66:39629)
0929 15:58:19.581054 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.581059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.582767 (+ 1708us) server_negotiation.cc:407] Connection header received
0929 15:58:19.582920 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.582924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.582991 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.583066 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:19.583582 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.584312 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.586052 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.586709 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.586784 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.587289 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:19.587406 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:19.587508 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.587640 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.587722 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 15:58:19.588089 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.580459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:39629 (local address 127.23.222.67:47287)
0929 15:58:19.582635 (+ 2176us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:19.582657 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:19.582794 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:19.583117 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:19.583122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:19.583136 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 15:58:19.583433 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.583446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.584467 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.584474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.585856 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:19.585869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.586920 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.586924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:19.587043 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.587075 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 15:58:19.587511 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 15:58:19.587526 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 15:58:19.587610 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 15:58:19.587749 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2020,"thread_start_us":1888,"threads_started":1}
I20260929 15:58:19.639114 32032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:19.639425 32032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:19.643198 32032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4f681fe561d433aa3ec834a455133d8: Sending DeleteTablet for 3 replicas of tablet a8dee00ca21f44c6a25c61e66ed16b20
I20260929 15:58:19.644500 32171 tablet_service.cc:1514] Processing DeleteTablet for tablet a8dee00ca21f44c6a25c61e66ed16b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:19 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:48548
I20260929 15:58:19.644838 32310 tablet_service.cc:1514] Processing DeleteTablet for tablet a8dee00ca21f44c6a25c61e66ed16b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:19 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:60226
I20260929 15:58:19.645201 32613 tablet_replica.cc:330] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: stopping tablet replica
I20260929 15:58:19.645418 32613 raft_consensus.cc:2229] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:19.645668 32613 raft_consensus.cc:2258] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:19.645965 32611 tablet_replica.cc:330] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: stopping tablet replica
I20260929 15:58:19.646153 32611 raft_consensus.cc:2229] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:19.646700 32611 raft_consensus.cc:2258] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:19.647790 32448 tablet_service.cc:1514] Processing DeleteTablet for tablet a8dee00ca21f44c6a25c61e66ed16b20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:19 UTC) from {username='oryx', principal='oryx/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:38158
I20260929 15:58:19.648149 32612 tablet_replica.cc:330] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: stopping tablet replica
I20260929 15:58:19.648355 32612 raft_consensus.cc:2229] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:19.648578 32612 raft_consensus.cc:2258] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:19.649147 32613 ts_tablet_manager.cc:1905] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:19.650204 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 32105
I20260929 15:58:19.657763 32612 ts_tablet_manager.cc:1905] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:19.662163 32611 ts_tablet_manager.cc:1905] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:19.678049 32613 ts_tablet_manager.cc:1918] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:19.678220 32613 log.cc:1196] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/a8dee00ca21f44c6a25c61e66ed16b20
I20260929 15:58:19.678627 32611 ts_tablet_manager.cc:1918] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:19.678696 32613 ts_tablet_manager.cc:1939] T a8dee00ca21f44c6a25c61e66ed16b20 P d1508a7e9e7e45beb71585854e4cd621: Deleting consensus metadata
I20260929 15:58:19.678891 32611 log.cc:1196] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/a8dee00ca21f44c6a25c61e66ed16b20
I20260929 15:58:19.679425 32611 ts_tablet_manager.cc:1939] T a8dee00ca21f44c6a25c61e66ed16b20 P 54e7b7f978974e91b37e9de8daac8ab6: Deleting consensus metadata
I20260929 15:58:19.680699 32018 catalog_manager.cc:4841] TS 54e7b7f978974e91b37e9de8daac8ab6 (127.23.222.66:39629): tablet a8dee00ca21f44c6a25c61e66ed16b20 (table test-table [id=5f43fa3335584f8b812440b0d746015c]) successfully deleted
I20260929 15:58:19.681483 32612 ts_tablet_manager.cc:1918] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 15:58:19.681630 32612 log.cc:1196] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/a8dee00ca21f44c6a25c61e66ed16b20
I20260929 15:58:19.682076 32612 ts_tablet_manager.cc:1939] T a8dee00ca21f44c6a25c61e66ed16b20 P 15b16f81004c449f994a872aae65466d: Deleting consensus metadata
I20260929 15:58:19.683154 32020 catalog_manager.cc:4841] TS 15b16f81004c449f994a872aae65466d (127.23.222.67:39081): tablet a8dee00ca21f44c6a25c61e66ed16b20 (table test-table [id=5f43fa3335584f8b812440b0d746015c]) successfully deleted
I20260929 15:58:19.696255 32018 catalog_manager.cc:4841] TS d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409): tablet a8dee00ca21f44c6a25c61e66ed16b20 (table test-table [id=5f43fa3335584f8b812440b0d746015c]) successfully deleted
Sep 29 15:58:19 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.65@KRBTEST.COM
I20260929 15:58:19.738061 32631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.651477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.23.222.65:35409)
0929 15:58:19.654645 (+ 3168us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.654656 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.697093 (+ 42437us) server_negotiation.cc:407] Connection header received
0929 15:58:19.697421 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.697427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.697528 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.697612 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:19.703476 (+ 5864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.704677 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.706728 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.707448 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.707554 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.718681 (+ 11127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:19.718698 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:19.718718 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:19.718780 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:19.723832 (+ 5052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.724925 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.724935 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.724940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.725137 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.735528 (+ 10391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.735536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.735543 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.736130 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:19.736479 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.737214 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.737530 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2982}
I20260929 15:58:19.739827 32236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:19.928704 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 32105
I20260929 15:58:19.951050 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 32245
Sep 29 15:58:19 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.66@KRBTEST.COM
I20260929 15:58:19.967408 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:19.952045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.23.222.66:39629)
0929 15:58:19.952260 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:19.952266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:19.952367 (+ 101us) server_negotiation.cc:407] Connection header received
0929 15:58:19.952634 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:19.952640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:19.952726 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:19.952868 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:19.954240 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.954973 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.956268 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:19.956718 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:19.956860 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:19.961086 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:19.961099 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:19.961115 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:19.961150 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:19.964379 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.965238 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.965250 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.965258 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.965404 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:19.965774 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:19.965780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:19.965783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:19.966143 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:19.966528 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:19.966768 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 15:58:19.966992 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 15:58:19.968261 32375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:20.028487 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:20.027700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:35409 (local address 127.23.222.67:57505)
0929 15:58:20.028174 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:20.028302 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":244,"threads_started":1}
W20260929 15:58:20.028802 32400 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.222.65:35409: connect: Connection refused (error 111)
I20260929 15:58:20.030267 32660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:20.029919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:35409 (local address 127.23.222.67:52121)
0929 15:58:20.030067 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:20.030147 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:35409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 15:58:20.030458 32400 consensus_peers.cc:487] T ca285cba245e40c299204a56808e5724 P 15b16f81004c449f994a872aae65466d -> Peer d1508a7e9e7e45beb71585854e4cd621 (127.23.222.65:35409): Couldn't send request to peer d1508a7e9e7e45beb71585854e4cd621. Status: Network error: Client connection negotiation failed: client connection to 127.23.222.65:35409: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 15:58:20.090374 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 32245
I20260929 15:58:20.110129 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 32384
Sep 29 15:58:20 dist-test-slave-cq88 krb5kdc[31989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.222.67@KRBTEST.COM
I20260929 15:58:20.125680 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:20.111035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.23.222.67:39081)
0929 15:58:20.111496 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:20.111500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:20.111514 (+ 14us) server_negotiation.cc:407] Connection header received
0929 15:58:20.111568 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:20.111571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:20.111627 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:20.111749 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:20.113053 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.113769 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.115071 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:20.115589 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:20.115729 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:20.120233 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:20.120249 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:20.120267 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:20.120325 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:20.123085 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:20.123952 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:20.123960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:20.123962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:20.124091 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:20.124423 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:20.124426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:20.124428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:20.124774 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:20.125014 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:20.125236 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 15:58:20.125412 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":256,"threads_started":1}
I20260929 15:58:20.126417 32514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:20.277931 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 32384
I20260929 15:58:20.303071 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 32002
I20260929 15:58:20.304445 32074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:20.419106 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 32002
2026-09-29T15:58:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6198 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:20 dist-test-slave-cq88 krb5kdc[32667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:20 dist-test-slave-cq88 krb5kdc[32667](info): set up 2 sockets
Sep 29 15:58:20 dist-test-slave-cq88 krb5kdc[32667](info): commencing operation
krb5kdc: starting...
W20260929 15:58:21.535207 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:58:21 dist-test-slave-cq88 krb5kdc[32667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:58:21.767928 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34405
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:46027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:34405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:22.108567 32680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:22.108883 32680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:22.109054 32680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:22.116461 32680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:22.116588 32680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:22.116679 32680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:22.116796 32680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:22.116889 32680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:22.127270 32680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:46027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:34405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:22.128901 32680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:22.130384 32680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:22.142764 32685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:22.142809 32688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:22.142709 32686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:22.143754 32680 server_base.cc:1034] running on GCE node
I20260929 15:58:22.144600 32680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:22.146100 32680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:22.147389 32680 hybrid_clock.cc:648] HybridClock initialized: now 1790697502147335 us; error 42 us; skew 500 ppm
I20260929 15:58:22.147796 32680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T15:58:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1795 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:22 dist-test-slave-cq88 krb5kdc[32694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:22 dist-test-slave-cq88 krb5kdc[32694](info): set up 2 sockets
Sep 29 15:58:22 dist-test-slave-cq88 krb5kdc[32694](info): commencing operation
krb5kdc: starting...
W20260929 15:58:23.315097 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:58:23 dist-test-slave-cq88 krb5kdc[32694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:58:23.548553 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:32943
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:32943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:23.876461 32707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:23.876782 32707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:23.876873 32707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:23.884162 32707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:23.884280 32707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:23.884326 32707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:23.884377 32707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:23.884423 32707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:23.894438 32707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:32943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:32943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:23.895802 32707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:23.897086 32707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:23.909045 32712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:23.909072 32713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:23.909006 32715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:23.909991 32707 server_base.cc:1034] running on GCE node
I20260929 15:58:23.910830 32707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:23.912374 32707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:23.913672 32707 hybrid_clock.cc:648] HybridClock initialized: now 1790697503913618 us; error 68 us; skew 500 ppm
I20260929 15:58:23.914077 32707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:23 dist-test-slave-cq88 krb5kdc[32694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:23.922201 32707 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:58:23.924574 32707 webserver.cc:460] Webserver started at http://127.23.222.126:44003/ using document root <none> and password file <none>
I20260929 15:58:23.925727 32707 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:23.925896 32707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:23.926303 32707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:23.930640 32707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "c6cde368c57a4b3f8cc934a5e21e3047"
format_stamp: "Formatted at 2026-09-29 15:58:23 on dist-test-slave-cq88"
server_key: "81b388f1a981ad93c03cae5868cd9bf0"
server_key_iv: "60bdb2caae1a0ee0b95653c502b53fac"
server_key_version: "encryptionkey@0"
I20260929 15:58:23.931571 32707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "c6cde368c57a4b3f8cc934a5e21e3047"
format_stamp: "Formatted at 2026-09-29 15:58:23 on dist-test-slave-cq88"
server_key: "81b388f1a981ad93c03cae5868cd9bf0"
server_key_iv: "60bdb2caae1a0ee0b95653c502b53fac"
server_key_version: "encryptionkey@0"
I20260929 15:58:23.938293 32707 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:58:23.942767 32722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:23.944344 32707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:23.944547 32707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "c6cde368c57a4b3f8cc934a5e21e3047"
format_stamp: "Formatted at 2026-09-29 15:58:23 on dist-test-slave-cq88"
server_key: "81b388f1a981ad93c03cae5868cd9bf0"
server_key_iv: "60bdb2caae1a0ee0b95653c502b53fac"
server_key_version: "encryptionkey@0"
I20260929 15:58:23.944782 32707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:23.975622 32707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:23.979925 32707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:23.980171 32707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:23.999034 32707 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:32943
I20260929 15:58:23.999070 305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:32943 every 8 connection(s)
I20260929 15:58:24.000535 32707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:24.001111 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 32707
I20260929 15:58:24.001385 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:58:24.002221 24441 external_mini_cluster.cc:1428] Setting key ab99a2db83ab87b9ea16847242e7b1da
I20260929 15:58:24.005504 306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 15:58:24 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:24.018667 306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: Bootstrap starting.
I20260929 15:58:24.022109 306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:24.023417 306 log.cc:824] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:24.026929 306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: No bootstrap required, opened a new log
I20260929 15:58:24.027333 309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.004453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51684 (local address 127.23.222.126:32943)
0929 15:58:24.005284 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.005300 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.005353 (+ 53us) server_negotiation.cc:407] Connection header received
0929 15:58:24.007757 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.007788 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.008309 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.008777 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:24.010138 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.012268 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.013703 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.014747 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.014939 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.019205 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:24.019214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:24.019230 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:24.019348 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:24.022579 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.023491 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.023498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.023505 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.023715 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.024228 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.024234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.024237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.025045 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:24.025255 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.025831 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.026220 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":315,"threads_started":1}
I20260929 15:58:24.035388 306 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } }
I20260929 15:58:24.035810 306 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:24.035933 306 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cde368c57a4b3f8cc934a5e21e3047, State: Initialized, Role: FOLLOWER
I20260929 15:58:24.036590 306 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } }
I20260929 15:58:24.036819 306 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:24.036972 306 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:24.037166 306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:24.039839 306 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } }
I20260929 15:58:24.040422 306 leader_election.cc:302] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6cde368c57a4b3f8cc934a5e21e3047; no voters:
I20260929 15:58:24.041083 306 leader_election.cc:288] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:24.041254 311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:24.041993 311 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 1 LEADER]: Becoming Leader. State: Replica: c6cde368c57a4b3f8cc934a5e21e3047, State: Running, Role: LEADER
I20260929 15:58:24.042613 311 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } }
I20260929 15:58:24.043356 306 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:24.045349 313 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6cde368c57a4b3f8cc934a5e21e3047. Latest consensus state: current_term: 1 leader_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } } }
I20260929 15:58:24.045742 313 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:24.046530 312 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } } }
I20260929 15:58:24.046823 312 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:24.047327 317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:24.052907 317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:24.064852 317 catalog_manager.cc:1347] Generated new cluster ID: 230d92ee2a474067a32d0221cecb3d22
I20260929 15:58:24.065006 317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:24.092880 317 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:24.094332 317 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:24.104038 317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: Generated new TSK 0
I20260929 15:58:24.105185 317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:58:24.163918 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:32943
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:24.514403 332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:24.514789 332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:24.514972 332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:24.522521 332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:24.522693 332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:24.522895 332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:58:24.533535 332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:32943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:24.535030 332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:24.536477 332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:24.551173 338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:24.551144 337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:24.551604 340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:24.552055 332 server_base.cc:1034] running on GCE node
I20260929 15:58:24.552678 332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:24.553709 332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:24.554957 332 hybrid_clock.cc:648] HybridClock initialized: now 1790697504554909 us; error 46 us; skew 500 ppm
I20260929 15:58:24.555377 332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:24 dist-test-slave-cq88 krb5kdc[32694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:24.559103 332 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:58:24.560847 332 webserver.cc:460] Webserver started at http://127.23.222.65:43585/ using document root <none> and password file <none>
I20260929 15:58:24.561595 332 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:24.561735 332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:24.562070 332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:24.565201 332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "5f1a2c0de18043ef95085627f4267c5f"
format_stamp: "Formatted at 2026-09-29 15:58:24 on dist-test-slave-cq88"
server_key: "239ca07be5deaf1138f71ba6ea4397fe"
server_key_iv: "bf7ad4ee24771e4936c75bdf3eef5d42"
server_key_version: "encryptionkey@0"
I20260929 15:58:24.566071 332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "5f1a2c0de18043ef95085627f4267c5f"
format_stamp: "Formatted at 2026-09-29 15:58:24 on dist-test-slave-cq88"
server_key: "239ca07be5deaf1138f71ba6ea4397fe"
server_key_iv: "bf7ad4ee24771e4936c75bdf3eef5d42"
server_key_version: "encryptionkey@0"
I20260929 15:58:24.571995 332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 15:58:24.575882 347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:24.577369 332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:58:24.577574 332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "5f1a2c0de18043ef95085627f4267c5f"
format_stamp: "Formatted at 2026-09-29 15:58:24 on dist-test-slave-cq88"
server_key: "239ca07be5deaf1138f71ba6ea4397fe"
server_key_iv: "bf7ad4ee24771e4936c75bdf3eef5d42"
server_key_version: "encryptionkey@0"
I20260929 15:58:24.577817 332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:24.590291 332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:24.594640 332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:24.594971 332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:24.595978 332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:24.597339 332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:24.597443 332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:24.597594 332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:24.597666 332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:24.631889 332 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:45591
I20260929 15:58:24.632000 460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:45591 every 8 connection(s)
I20260929 15:58:24.633246 332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:24.642396 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 332
I20260929 15:58:24.642612 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:24.643299 24441 external_mini_cluster.cc:1428] Setting key 09b68a51cff4853b12dd318cc069bdd4
Sep 29 15:58:24 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697504, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:24.654974 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.636032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:45125 (local address 127.23.222.126:32943)
0929 15:58:24.636517 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:24.636524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:24.637648 (+ 1124us) server_negotiation.cc:407] Connection header received
0929 15:58:24.639627 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:24.639632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:24.639713 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:24.639821 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:24.641494 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.642222 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.644466 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.645060 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.645182 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.649108 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:24.649113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:24.649116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:24.649149 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:24.651743 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.652520 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.652527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.652536 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.652651 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:24.653137 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:24.653142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:24.653145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:24.653446 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:24.653602 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:24.654374 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 15:58:24.654609 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":236,"threads_started":1}
I20260929 15:58:24.656041 464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:24.636668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:45125)
0929 15:58:24.637446 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:24.637510 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:24.639193 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:24.640059 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:24.640068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:24.640650 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:24.641279 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.641305 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.642432 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.642436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.644299 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:24.644342 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:24.645323 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:24.645331 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:24.645564 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:24.646266 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:24.646302 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:24.648861 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:24.651900 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:24.651911 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:24.651930 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:24.652386 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:24.652801 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:24.652806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:24.652809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:24.653015 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:24.653581 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:24.653589 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:24.653969 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 15:58:24.654243 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 15:58:24.654809 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":310,"threads_started":1}
I20260929 15:58:24.657866 461 heartbeater.cc:347] Connected to a master server at 127.23.222.126:32943
I20260929 15:58:24.658205 461 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:24.659060 461 heartbeater.cc:511] Master 127.23.222.126:32943 requested a full tablet report, sending...
I20260929 15:58:24.661377 32739 ts_manager.cc:194] Registered new tserver with Master: 5f1a2c0de18043ef95085627f4267c5f (127.23.222.65:45591)
I20260929 15:58:24.662878 32739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:45125
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:58:24.695473 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:32943
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:25.041704 467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:25.042001 467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:25.042163 467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:25.050064 467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:25.050182 467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:25.050385 467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:58:25.061088 467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:32943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:25.062441 467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:25.063917 467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:25.078244 476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:25.078271 473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:25.078331 474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:25.079737 467 server_base.cc:1034] running on GCE node
I20260929 15:58:25.080417 467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:25.081501 467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:25.082749 467 hybrid_clock.cc:648] HybridClock initialized: now 1790697505082688 us; error 51 us; skew 500 ppm
I20260929 15:58:25.083115 467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:25 dist-test-slave-cq88 krb5kdc[32694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:25.086578 467 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:58:25.088371 467 webserver.cc:460] Webserver started at http://127.23.222.66:45335/ using document root <none> and password file <none>
I20260929 15:58:25.089128 467 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:25.089277 467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:25.089608 467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:25.092777 467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "c60012f4f5214f569b47a05732dcc37a"
format_stamp: "Formatted at 2026-09-29 15:58:25 on dist-test-slave-cq88"
server_key: "ddcf8d13c6c8d4a2d47b78cffac5402a"
server_key_iv: "fd2d60f4c89b0caad4963c4281599403"
server_key_version: "encryptionkey@0"
I20260929 15:58:25.093622 467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "c60012f4f5214f569b47a05732dcc37a"
format_stamp: "Formatted at 2026-09-29 15:58:25 on dist-test-slave-cq88"
server_key: "ddcf8d13c6c8d4a2d47b78cffac5402a"
server_key_iv: "fd2d60f4c89b0caad4963c4281599403"
server_key_version: "encryptionkey@0"
I20260929 15:58:25.099200 467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:58:25.102721 483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:25.104169 467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:25.104350 467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "c60012f4f5214f569b47a05732dcc37a"
format_stamp: "Formatted at 2026-09-29 15:58:25 on dist-test-slave-cq88"
server_key: "ddcf8d13c6c8d4a2d47b78cffac5402a"
server_key_iv: "fd2d60f4c89b0caad4963c4281599403"
server_key_version: "encryptionkey@0"
I20260929 15:58:25.104560 467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:25.130663 467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:25.135437 467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:25.135761 467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:25.136757 467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:25.138132 467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:25.138237 467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:25.138394 467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:25.138468 467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:25.172780 467 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:43209
I20260929 15:58:25.172816 596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:43209 every 8 connection(s)
I20260929 15:58:25.174132 467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:25.179219 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 467
I20260929 15:58:25.179431 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:25.179981 24441 external_mini_cluster.cc:1428] Setting key f7e5a739ece2fe88fe5152e5d0ef6a00
Sep 29 15:58:25 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:25.198426 599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.176634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:57501 (local address 127.23.222.126:32943)
0929 15:58:25.177087 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:25.177091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:25.177984 (+ 893us) server_negotiation.cc:407] Connection header received
0929 15:58:25.180354 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:25.180359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:25.180435 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:25.180536 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:25.182459 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.183366 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.185941 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.186403 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.186516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:25.191152 (+ 4636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:25.191162 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:25.191170 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:25.191219 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:25.194881 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.195707 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.195715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.195722 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.195879 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.196365 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.196372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.196378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.196787 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:25.196945 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:25.197683 (+ 738us) server_negotiation.cc:299] Negotiation successful
0929 15:58:25.197948 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":258,"threads_started":1}
I20260929 15:58:25.199090 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.177097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:57501)
0929 15:58:25.177806 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:25.177856 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:25.180033 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:25.180858 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:25.180872 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:25.181484 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:25.182278 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:25.182303 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.183547 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.183556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:25.185759 (+ 2203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:25.185776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.186645 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.186653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:25.186984 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:25.187747 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:25.187783 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:25.190869 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:25.195043 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:25.195051 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:25.195066 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:25.195577 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:25.196019 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:25.196023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:25.196025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:25.196232 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:25.196939 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:25.196945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:25.197279 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 15:58:25.197506 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 15:58:25.197930 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":253,"threads_started":1}
I20260929 15:58:25.200711 597 heartbeater.cc:347] Connected to a master server at 127.23.222.126:32943
I20260929 15:58:25.201092 597 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:25.201846 597 heartbeater.cc:511] Master 127.23.222.126:32943 requested a full tablet report, sending...
I20260929 15:58:25.203517 32739 ts_manager.cc:194] Registered new tserver with Master: c60012f4f5214f569b47a05732dcc37a (127.23.222.66:43209)
I20260929 15:58:25.204429 32739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:57501
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:58:25.235854 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:32943
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:25.572224 603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:25.572623 603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:25.572779 603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:25.580411 603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:25.580525 603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:25.580641 603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:58:25.590482 603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:32943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:25.591825 603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:25.593108 603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:25.606146 608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:25.606184 611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:25.606127 609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:25.607400 603 server_base.cc:1034] running on GCE node
I20260929 15:58:25.608044 603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:25.609169 603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:25.610394 603 hybrid_clock.cc:648] HybridClock initialized: now 1790697505610340 us; error 53 us; skew 500 ppm
I20260929 15:58:25.610733 603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:25 dist-test-slave-cq88 krb5kdc[32694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:25.614089 603 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:58:25.615694 603 webserver.cc:460] Webserver started at http://127.23.222.67:35863/ using document root <none> and password file <none>
I20260929 15:58:25.616413 603 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:25.616561 603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:25.616886 603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:25.620018 603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "ce9c7986d7384174b1959f9dede8452d"
format_stamp: "Formatted at 2026-09-29 15:58:25 on dist-test-slave-cq88"
server_key: "0ca3dfffbcb350da88b81b319e773d90"
server_key_iv: "99546fffcc47cd2566369a7f2ed47bd3"
server_key_version: "encryptionkey@0"
I20260929 15:58:25.620865 603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "ce9c7986d7384174b1959f9dede8452d"
format_stamp: "Formatted at 2026-09-29 15:58:25 on dist-test-slave-cq88"
server_key: "0ca3dfffbcb350da88b81b319e773d90"
server_key_iv: "99546fffcc47cd2566369a7f2ed47bd3"
server_key_version: "encryptionkey@0"
I20260929 15:58:25.626394 603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:58:25.630317 618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:25.632213 603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:58:25.632403 603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "ce9c7986d7384174b1959f9dede8452d"
format_stamp: "Formatted at 2026-09-29 15:58:25 on dist-test-slave-cq88"
server_key: "0ca3dfffbcb350da88b81b319e773d90"
server_key_iv: "99546fffcc47cd2566369a7f2ed47bd3"
server_key_version: "encryptionkey@0"
I20260929 15:58:25.632697 603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:25.654533 603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:25.659443 603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:25.659737 603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:25.660830 603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:25.662797 603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:25.662886 603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:25.662990 603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:25.663043 603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:25.666332 461 heartbeater.cc:503] Master 127.23.222.126:32943 was elected leader, sending a full tablet report...
I20260929 15:58:25.698289 603 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:36753
I20260929 15:58:25.698336 731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:36753 every 8 connection(s)
I20260929 15:58:25.699791 603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:25.709605 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 603
I20260929 15:58:25.709828 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:25.710376 24441 external_mini_cluster.cc:1428] Setting key 2689f5d596997af0a292311bb45d17ba
Sep 29 15:58:25 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697505, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:25.721278 734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.702496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:35131 (local address 127.23.222.126:32943)
0929 15:58:25.702995 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:25.703001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:25.704141 (+ 1140us) server_negotiation.cc:407] Connection header received
0929 15:58:25.705971 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:25.705976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:25.706056 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:25.706173 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:25.707799 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.708654 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.711166 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.711653 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.711801 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:25.715518 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:25.715523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:25.715526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:25.715563 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:25.718068 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.718902 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.718915 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.718918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.719042 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.719622 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.719628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.719631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.719927 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:25.720053 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:25.720795 (+ 742us) server_negotiation.cc:299] Negotiation successful
0929 15:58:25.720977 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":230,"threads_started":1}
I20260929 15:58:25.722270 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.703127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:35131)
0929 15:58:25.703953 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:25.704010 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:25.705692 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:25.706388 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:25.706397 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:25.706953 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:25.707618 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:25.707636 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.708905 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.708909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:25.711003 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:25.711017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.711902 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.711910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:25.712154 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:25.712862 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:25.712891 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:25.715282 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:25.718242 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:25.718252 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:25.718271 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:25.718763 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:25.719189 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:25.719192 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:25.719195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:25.719480 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:25.720056 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:25.720062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:25.720433 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 15:58:25.720674 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 15:58:25.721084 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":325,"threads_started":1}
I20260929 15:58:25.723848 732 heartbeater.cc:347] Connected to a master server at 127.23.222.126:32943
I20260929 15:58:25.724187 732 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:25.725096 732 heartbeater.cc:511] Master 127.23.222.126:32943 requested a full tablet report, sending...
I20260929 15:58:25.726643 32739 ts_manager.cc:194] Registered new tserver with Master: ce9c7986d7384174b1959f9dede8452d (127.23.222.67:36753)
I20260929 15:58:25.727506 32739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:35131
I20260929 15:58:25.738971 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab.
Entry for principal oryx/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab.
WARNING: no policy specified for oryx/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab.
Entry for principal oryx/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab.
WARNING: no policy specified for oryx/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab.
Entry for principal oryx/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab.
I20260929 15:58:25.887130 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 332
Sep 29 15:58:25 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:58:25.904094 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:25.888471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.23.222.65:45591)
0929 15:58:25.889240 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:25.889248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:25.889276 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:58:25.889337 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:25.889343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:25.889557 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:25.889689 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:25.891125 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.892245 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.893538 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:25.894148 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:25.894247 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:25.898463 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:25.898476 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:25.898491 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:25.898531 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:25.901441 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.902139 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.902146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.902148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.902297 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:25.902638 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:25.902641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:25.902644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:25.903021 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:25.903325 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:25.903559 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 15:58:25.903799 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":421,"threads_started":1}
I20260929 15:58:25.905460 456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:25.990329 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 332
I20260929 15:58:26.007128 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 467
Sep 29 15:58:26 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:58:26.024701 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.008256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.23.222.66:43209)
0929 15:58:26.008826 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.008833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.008859 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:58:26.008917 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.008923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.009134 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.009323 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:26.010615 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.011719 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.013016 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.013616 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.013783 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.018437 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:26.018452 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:26.018469 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:26.018507 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:26.021581 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:26.022360 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:26.022368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:26.022370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:26.022515 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:26.022982 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:26.022986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:26.022988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:26.023544 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:26.023826 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.024131 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.024372 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":322,"threads_started":1}
I20260929 15:58:26.026139 592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:26.109856 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 467
I20260929 15:58:26.126130 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 603
Sep 29 15:58:26 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:58:26.142926 747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:26.127219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.23.222.67:36753)
0929 15:58:26.127896 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:26.127903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:26.127932 (+ 29us) server_negotiation.cc:407] Connection header received
0929 15:58:26.127995 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:26.128002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:26.128210 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:26.128345 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:26.129687 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.130623 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.132050 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:26.132591 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:26.132689 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:26.137072 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:26.137083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:26.137098 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:26.137130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:26.140360 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:26.141014 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:26.141022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:26.141024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:26.141155 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:26.141520 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:26.141524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:26.141526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:26.141923 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:26.142160 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:26.142412 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 15:58:26.142601 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":337,"threads_started":1}
I20260929 15:58:26.144506 727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:26.232249 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 603
I20260929 15:58:26.248838 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 32707
I20260929 15:58:26.250151 301 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:26.353286 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 32707
I20260929 15:58:26.374703 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:32943
--webserver_interface=127.23.222.126
--webserver_port=44003
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:32943
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:26.707630 750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:26.707978 750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:26.708139 750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:26.716401 750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:26.716533 750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:26.716634 750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:26.716742 750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:26.716815 750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:26.727147 750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:32943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:32943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=44003
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:26.728606 750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:26.729974 750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:26.743151 755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:26.743126 756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:26.743072 758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:26.749560 750 server_base.cc:1034] running on GCE node
I20260929 15:58:26.750761 750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:26.752553 750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:26.753800 750 hybrid_clock.cc:648] HybridClock initialized: now 1790697506753753 us; error 51 us; skew 500 ppm
I20260929 15:58:26.754168 750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:26 dist-test-slave-cq88 krb5kdc[32694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697506, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:26.757479 750 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:58:26.759155 750 webserver.cc:460] Webserver started at http://127.23.222.126:44003/ using document root <none> and password file <none>
I20260929 15:58:26.760094 750 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:26.760241 750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:26.766302 750 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 15:58:26.769387 765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:26.770880 750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:26.771083 750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "c6cde368c57a4b3f8cc934a5e21e3047"
format_stamp: "Formatted at 2026-09-29 15:58:23 on dist-test-slave-cq88"
server_key: "81b388f1a981ad93c03cae5868cd9bf0"
server_key_iv: "60bdb2caae1a0ee0b95653c502b53fac"
server_key_version: "encryptionkey@0"
I20260929 15:58:26.772158 750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:26.806473 750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:26.813010 750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:26.813455 750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:26.833760 750 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:32943
I20260929 15:58:26.833806 816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:32943 every 8 connection(s)
I20260929 15:58:26.835444 750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:26.838014 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 750
I20260929 15:58:26.839020 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:45591
--local_ip_for_outbound_sockets=127.23.222.65
--tserver_master_addrs=127.23.222.126:32943
--webserver_port=43585
--webserver_interface=127.23.222.65
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.222.65
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:58:26.845620 817 sys_catalog.cc:263] Verifying existing consensus state
I20260929 15:58:26.849917 817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: Bootstrap starting.
I20260929 15:58:26.870101 817 log.cc:824] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:26.879781 817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 15:58:26.880362 817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: Bootstrap complete.
I20260929 15:58:26.889864 817 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } }
I20260929 15:58:26.890727 817 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cde368c57a4b3f8cc934a5e21e3047, State: Initialized, Role: FOLLOWER
I20260929 15:58:26.891489 817 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } }
I20260929 15:58:26.891754 817 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:26.891870 817 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:26.892063 817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 1 FOLLOWER]: Advancing to term 2
I20260929 15:58:26.896286 817 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } }
I20260929 15:58:26.896915 817 leader_election.cc:302] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6cde368c57a4b3f8cc934a5e21e3047; no voters:
I20260929 15:58:26.897620 817 leader_election.cc:288] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 15:58:26.897759 821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 15:58:26.898990 821 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [term 2 LEADER]: Becoming Leader. State: Replica: c6cde368c57a4b3f8cc934a5e21e3047, State: Running, Role: LEADER
I20260929 15:58:26.899607 821 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } }
I20260929 15:58:26.900002 817 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:26.902276 822 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } } }
I20260929 15:58:26.902773 822 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:26.906605 823 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6cde368c57a4b3f8cc934a5e21e3047. Latest consensus state: current_term: 2 leader_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6cde368c57a4b3f8cc934a5e21e3047" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 32943 } } }
I20260929 15:58:26.907126 823 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:26.908893 827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:26.915798 827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:26.917734 827 catalog_manager.cc:1259] Loaded cluster ID: 230d92ee2a474067a32d0221cecb3d22
I20260929 15:58:26.917861 827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:26.923379 827 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:26.925385 827 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c6cde368c57a4b3f8cc934a5e21e3047: Loaded TSK: 0
I20260929 15:58:26.927495 827 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 15:58:27.206671 819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:27.206979 819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:27.207131 819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:27.214970 819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:27.215093 819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:27.215337 819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:58:27.226239 819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.65.keytab
--principal=oryx/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:45591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=43585
--tserver_master_addrs=127.23.222.126:32943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:27.227785 819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:27.229133 819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:27.244612 845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:27.244932 847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:27.245208 844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:27.245569 819 server_base.cc:1034] running on GCE node
I20260929 15:58:27.246308 819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:27.247560 819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:27.248802 819 hybrid_clock.cc:648] HybridClock initialized: now 1790697507248753 us; error 38 us; skew 500 ppm
I20260929 15:58:27.249171 819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697507, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:27.253178 819 init.cc:377] Logged in from keytab as oryx/127.23.222.65@KRBTEST.COM (short username oryx)
I20260929 15:58:27.254853 819 webserver.cc:460] Webserver started at http://127.23.222.65:43585/ using document root <none> and password file <none>
I20260929 15:58:27.255707 819 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:27.255877 819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:27.262203 819 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 15:58:27.265535 854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:27.267226 819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:27.267530 819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "5f1a2c0de18043ef95085627f4267c5f"
format_stamp: "Formatted at 2026-09-29 15:58:24 on dist-test-slave-cq88"
server_key: "239ca07be5deaf1138f71ba6ea4397fe"
server_key_iv: "bf7ad4ee24771e4936c75bdf3eef5d42"
server_key_version: "encryptionkey@0"
I20260929 15:58:27.268795 819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:27.297076 819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:27.302060 819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:27.302366 819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:27.303597 819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:27.305096 819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:27.305183 819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:27.305310 819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:27.305408 819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:27.338675 819 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:45591
I20260929 15:58:27.338797 966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:45591 every 8 connection(s)
I20260929 15:58:27.340304 819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:27.346292 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 819
I20260929 15:58:27.347091 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:43209
--local_ip_for_outbound_sockets=127.23.222.66
--tserver_master_addrs=127.23.222.126:32943
--webserver_port=45335
--webserver_interface=127.23.222.66
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.222.66
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:27.360845 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:27.343749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:56103)
0929 15:58:27.344656 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:27.344702 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:27.346493 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:27.348469 (+ 1976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:27.348477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:27.349052 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:27.349622 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.349641 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.352045 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.352049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.353356 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.353370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.354505 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.354513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.354814 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.355600 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:27.355632 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:27.359405 (+ 3773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":435,"threads_started":1}
W20260929 15:58:27.360877 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:27.343595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:56103 (local address 127.23.222.126:32943)
0929 15:58:27.344430 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:27.344448 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:27.344852 (+ 404us) server_negotiation.cc:407] Connection header received
0929 15:58:27.347220 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:27.347265 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:27.347805 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:27.348294 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:27.349843 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.351838 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.353522 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.354227 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.354462 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.359818 (+ 5356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:56103: BlockingRecv error: recv got EOF from 127.23.222.65:56103 (error 108)
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":258,"threads_started":1}
W20260929 15:58:27.361631 967 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:32943 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.222.126:32943: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:27.372851 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:27.362294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:37459)
0929 15:58:27.362528 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:27.362550 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:27.362676 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:27.363046 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:27.363053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:27.363623 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:27.363870 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.363885 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.365039 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.365045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.366808 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.366829 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.367688 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.367697 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.367932 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.368568 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:27.368599 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:27.372549 (+ 3950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260929 15:58:27.373265 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:27.362493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:37459 (local address 127.23.222.126:32943)
0929 15:58:27.362656 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:27.362681 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:27.362705 (+ 24us) server_negotiation.cc:407] Connection header received
0929 15:58:27.362808 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:27.362815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:27.362892 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:27.362976 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:27.364069 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.364862 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.366997 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.367494 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.367636 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.372854 (+ 5218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:37459: BlockingRecv error: recv got EOF from 127.23.222.65:37459 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:27.383225 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:27.373888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:42375)
0929 15:58:27.374072 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:27.374091 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:27.374193 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:27.374544 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:27.374549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:27.374924 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:27.375118 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.375128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.376349 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.376354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.377882 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.377898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.378592 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.378598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.378778 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.379462 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:27.379486 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:27.382924 (+ 3438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 15:58:27.383584 967 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:58:27.383765 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:27.374085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:42375 (local address 127.23.222.126:32943)
0929 15:58:27.374267 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:27.374271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:27.374289 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:58:27.374338 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:27.374342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:27.374411 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:27.374499 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:27.375334 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.376206 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.378032 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.378441 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.378548 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.383303 (+ 4755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:42375: BlockingRecv error: recv got EOF from 127.23.222.65:42375 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 15:58:27.688862 972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:27.689155 972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:27.689330 972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:27.696743 972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:27.696871 972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:27.697065 972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:58:27.707212 972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.66.keytab
--principal=oryx/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:43209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=45335
--tserver_master_addrs=127.23.222.126:32943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:27.709084 972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:27.710476 972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:27.724711 978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:27.724748 977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:27.724748 980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:27.725999 972 server_base.cc:1034] running on GCE node
I20260929 15:58:27.726568 972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:27.727620 972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:27.728837 972 hybrid_clock.cc:648] HybridClock initialized: now 1790697507728791 us; error 36 us; skew 500 ppm
I20260929 15:58:27.729174 972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697507, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:27.732451 972 init.cc:377] Logged in from keytab as oryx/127.23.222.66@KRBTEST.COM (short username oryx)
I20260929 15:58:27.734069 972 webserver.cc:460] Webserver started at http://127.23.222.66:45335/ using document root <none> and password file <none>
I20260929 15:58:27.734772 972 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:27.734911 972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:27.740579 972 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 15:58:27.743494 987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:27.744978 972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 15:58:27.745173 972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "c60012f4f5214f569b47a05732dcc37a"
format_stamp: "Formatted at 2026-09-29 15:58:25 on dist-test-slave-cq88"
server_key: "ddcf8d13c6c8d4a2d47b78cffac5402a"
server_key_iv: "fd2d60f4c89b0caad4963c4281599403"
server_key_version: "encryptionkey@0"
I20260929 15:58:27.746168 972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:27.762741 972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:27.767437 972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:27.767756 972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:27.768790 972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:27.770167 972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:27.770295 972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:27.770421 972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:27.770510 972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:27.805133 972 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:43209
I20260929 15:58:27.805225 1099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:43209 every 8 connection(s)
I20260929 15:58:27.806663 972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:27.812175 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 972
I20260929 15:58:27.812927 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:36753
--local_ip_for_outbound_sockets=127.23.222.67
--tserver_master_addrs=127.23.222.126:32943
--webserver_port=35863
--webserver_interface=127.23.222.67
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.222.67
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:27.826697 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:27.809677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:47507 (local address 127.23.222.126:32943)
0929 15:58:27.809903 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:27.809909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:27.811211 (+ 1302us) server_negotiation.cc:407] Connection header received
0929 15:58:27.814066 (+ 2855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:27.814072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:27.814152 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:27.814274 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:27.816573 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.817287 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.819998 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.820709 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.820866 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.826381 (+ 5515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:47507: BlockingRecv error: recv got EOF from 127.23.222.66:47507 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 15:58:27.827354 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:27.810207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:47507)
0929 15:58:27.811020 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:27.811062 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:27.813704 (+ 2642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:27.814606 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:27.814622 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:27.815490 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:27.816355 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.816385 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.817476 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.817485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.819799 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.819820 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.820949 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.820954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.821168 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.821953 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:27.821981 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:27.826089 (+ 4108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":365,"threads_started":1}
W20260929 15:58:27.828225 1100 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:32943 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.222.126:32943: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:27.840014 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:27.828906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:41929)
0929 15:58:27.829198 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:27.829221 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:27.829342 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:27.829674 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:27.829682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:27.830168 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:27.830384 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.830399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.831637 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.831645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.833428 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.833448 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.834531 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.834540 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.834785 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.835530 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:27.835568 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:27.839728 (+ 4160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 15:58:27.840425 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:27.829071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:41929 (local address 127.23.222.126:32943)
0929 15:58:27.829263 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:27.829268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:27.829286 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:58:27.829465 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:27.829469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:27.829529 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:27.829635 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:27.830579 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.831477 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.833618 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.834337 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.834496 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.839998 (+ 5502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:41929: BlockingRecv error: recv got EOF from 127.23.222.66:41929 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:27 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:27.851847 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:27.840831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:49489)
0929 15:58:27.842529 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:27.842551 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:27.842689 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:27.843101 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:27.843106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:27.843568 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:27.843802 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.843817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.845353 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.845360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.846505 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:27.846520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.847574 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.847579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:27.847767 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.848442 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:27.848460 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:27.851613 (+ 3153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1550}
W20260929 15:58:27.852180 1100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 15:58:27.852239 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:27.840957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:49489 (local address 127.23.222.126:32943)
0929 15:58:27.841172 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:27.841179 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:27.842645 (+ 1466us) server_negotiation.cc:407] Connection header received
0929 15:58:27.842820 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:27.842826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:27.842925 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:27.843031 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:27.843967 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.845180 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.846665 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:27.847382 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:27.847517 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:27.851878 (+ 4361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:49489: BlockingRecv error: recv got EOF from 127.23.222.66:49489 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 15:58:28.156426 1103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:28.156740 1103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:28.156834 1103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:28.164256 1103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:28.164378 1103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:28.164492 1103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:58:28.174935 1103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/krb5kdc/oryx_127.23.222.67.keytab
--principal=oryx/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:36753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=35863
--tserver_master_addrs=127.23.222.126:32943
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:28.176445 1103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:28.177881 1103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:28.192229 1108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:28.193315 1109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:28.194361 1111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:28.194577 1103 server_base.cc:1034] running on GCE node
I20260929 15:58:28.195325 1103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:28.196350 1103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:28.197559 1103 hybrid_clock.cc:648] HybridClock initialized: now 1790697508197506 us; error 51 us; skew 500 ppm
I20260929 15:58:28.197917 1103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697508, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:28.201282 1103 init.cc:377] Logged in from keytab as oryx/127.23.222.67@KRBTEST.COM (short username oryx)
I20260929 15:58:28.202914 1103 webserver.cc:460] Webserver started at http://127.23.222.67:35863/ using document root <none> and password file <none>
I20260929 15:58:28.203745 1103 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:28.203864 1103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:28.210179 1103 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260929 15:58:28.213392 1118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:28.215042 1103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:28.215337 1103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "ce9c7986d7384174b1959f9dede8452d"
format_stamp: "Formatted at 2026-09-29 15:58:25 on dist-test-slave-cq88"
server_key: "0ca3dfffbcb350da88b81b319e773d90"
server_key_iv: "99546fffcc47cd2566369a7f2ed47bd3"
server_key_version: "encryptionkey@0"
I20260929 15:58:28.216413 1103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:28.228607 1103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:28.233071 1103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:28.233388 1103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:28.234447 1103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:28.236042 1103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:28.236168 1103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:28.236294 1103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:28.236379 1103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:28.270504 1103 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:36753
I20260929 15:58:28.270596 1232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:36753 every 8 connection(s)
I20260929 15:58:28.272086 1103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:28.279842 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 1103
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:28.289463 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:28.274728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:34135 (local address 127.23.222.126:32943)
0929 15:58:28.274913 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:28.274918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:28.276161 (+ 1243us) server_negotiation.cc:407] Connection header received
0929 15:58:28.278110 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:28.278115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:28.278182 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:28.278272 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:28.279952 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.280873 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.282902 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.283607 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.283711 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.289184 (+ 5473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:34135: BlockingRecv error: recv got EOF from 127.23.222.67:34135 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 15:58:28.290288 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:28.275167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:34135)
0929 15:58:28.276007 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:28.276045 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:28.277864 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:28.278497 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:28.278505 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:28.279012 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:28.279784 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.279804 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.281055 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.281064 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.282746 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.282766 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.283856 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.283864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.284146 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.284878 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:28.284902 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:28.288957 (+ 4055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":300,"threads_started":1}
W20260929 15:58:28.291033 1233 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:32943 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.222.126:32943: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
I20260929 15:58:28.298307 1237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:28.280903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.23.222.126:32943)
0929 15:58:28.281694 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:28.281699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:28.281717 (+ 18us) server_negotiation.cc:407] Connection header received
0929 15:58:28.281763 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:28.281768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:28.281836 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:28.281945 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:28.283559 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.284706 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.286262 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.286811 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.286965 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.289985 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:28.289993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:28.290011 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:28.290055 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:28.294403 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:28.295395 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:28.295403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:28.295410 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:28.295606 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:28.296095 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:28.296101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:28.296104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:28.296766 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:28.297076 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:28.297590 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 15:58:28.298017 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":547,"threads_started":1}
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 15:58:28.301353 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:28.291898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:43165 (local address 127.23.222.126:32943)
0929 15:58:28.292043 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:28.292047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:28.292235 (+ 188us) server_negotiation.cc:407] Connection header received
0929 15:58:28.292417 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:28.292423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:28.292504 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:28.292629 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:28.293215 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.294459 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.296095 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.296737 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.296887 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.301099 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:43165: BlockingRecv error: recv got EOF from 127.23.222.67:43165 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 15:58:28.301672 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:28.291765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:43165)
0929 15:58:28.292135 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:28.292153 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:28.292290 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:28.292646 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:28.292649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:28.292943 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:28.293076 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.293083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.294600 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.294605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.295948 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.295964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.296911 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.296916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.297099 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.297692 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:28.297713 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:28.300868 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:28.309646 1235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:28.302342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:51299)
0929 15:58:28.302501 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:28.302520 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:28.302613 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:28.303012 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:28.303016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:28.303369 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:28.303515 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.303523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.304499 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.304503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.305397 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.305408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.306024 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.306028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.306143 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.306660 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:28.306676 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:28.309438 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 15:58:28.309940 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:28.302550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:51299 (local address 127.23.222.126:32943)
0929 15:58:28.302778 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:28.302782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:28.302796 (+ 14us) server_negotiation.cc:407] Connection header received
0929 15:58:28.302835 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:28.302839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:28.302890 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:28.302975 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:28.303649 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.304388 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.305516 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.305905 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.305999 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.309653 (+ 3654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:51299: BlockingRecv error: recv got EOF from 127.23.222.67:51299 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260929 15:58:28.310034 1233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:28.393741 1238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:28.384334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:52405)
0929 15:58:28.384941 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:28.384957 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:28.385133 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:28.385492 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:28.385496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:28.386005 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:28.386197 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.386206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.387230 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.387295 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.388345 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.388360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.389343 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.389348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.389549 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.390270 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:28.390288 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:28.393347 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":347,"threads_started":1}
W20260929 15:58:28.393872 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:28.384434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:52405 (local address 127.23.222.126:32943)
0929 15:58:28.384776 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:28.384783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:28.385058 (+ 275us) server_negotiation.cc:407] Connection header received
0929 15:58:28.385267 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:28.385278 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:28.385350 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:28.385458 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:28.386352 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.387072 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.388521 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.389144 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.389314 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.393603 (+ 4289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:52405: BlockingRecv error: recv got EOF from 127.23.222.65:52405 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:28 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:28.862430 1239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:28.852852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:60971)
0929 15:58:28.853557 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:28.853574 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:28.853724 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:28.854100 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:28.854106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:28.854568 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:28.854770 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.854779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.855790 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.855795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.856840 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:28.856876 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.857711 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.857716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:28.857846 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.858542 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:28.858560 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:28.862073 (+ 3513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":419,"threads_started":1}
W20260929 15:58:28.862612 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:28.852928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:60971 (local address 127.23.222.126:32943)
0929 15:58:28.853124 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:28.853129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:28.853665 (+ 536us) server_negotiation.cc:407] Connection header received
0929 15:58:28.853845 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:28.853849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:28.853943 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:28.854045 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:28.854897 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.855636 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.857041 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:28.857484 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:28.857588 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:28.862379 (+ 4791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:60971: BlockingRecv error: recv got EOF from 127.23.222.66:60971 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 15:58:29 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:29 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:29.329217 1240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:29.310718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:38363)
0929 15:58:29.311368 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:29.311388 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:29.311525 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:29.311928 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:29.311932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:29.312463 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:29.312685 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:29.312695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.314148 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.314156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:29.315537 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:29.315552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.316274 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.316280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:29.316427 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:29.325400 (+ 8973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:29.325444 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:29.328978 (+ 3534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":360,"threads_started":1}
W20260929 15:58:29.329468 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:29.310825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:38363 (local address 127.23.222.126:32943)
0929 15:58:29.310999 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:29.311004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:29.311470 (+ 466us) server_negotiation.cc:407] Connection header received
0929 15:58:29.311691 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:29.311696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:29.311768 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:29.311870 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:29.312857 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.313940 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.315683 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.316075 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.316180 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:29.329227 (+ 13047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:38363: BlockingRecv error: recv got EOF from 127.23.222.67:38363 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 15:58:29 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:29 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:29.403322 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:29.394824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:55575)
0929 15:58:29.395426 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:29.395442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:29.395556 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:29.395895 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:29.395898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:29.396316 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:29.396483 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:29.396491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.397492 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.397496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:29.398398 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:29.398409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.399065 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.399069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:29.399180 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:29.399855 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:29.399871 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:29.403025 (+ 3154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":324,"threads_started":1}
W20260929 15:58:29.403502 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:29.394866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:55575 (local address 127.23.222.126:32943)
0929 15:58:29.395030 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:29.395035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:29.395519 (+ 484us) server_negotiation.cc:407] Connection header received
0929 15:58:29.395654 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:29.395658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:29.395744 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:29.395835 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:29.396621 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.397369 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.398526 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.398928 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.399028 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:29.403212 (+ 4184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:55575: BlockingRecv error: recv got EOF from 127.23.222.65:55575 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 15:58:29 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:29 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:29.875774 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:29.863699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:42859)
0929 15:58:29.864214 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:29.864230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:29.864409 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:29.864828 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:29.865035 (+ 207us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:29.865838 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:29.866062 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:29.866073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.867444 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.867449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:29.869138 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:29.869160 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.870224 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.870229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:29.870395 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:29.871335 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:29.871368 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:29.875455 (+ 4087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":257,"threads_started":1}
W20260929 15:58:29.876116 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:29.863801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:42859 (local address 127.23.222.126:32943)
0929 15:58:29.864038 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:29.864045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:29.864328 (+ 283us) server_negotiation.cc:407] Connection header received
0929 15:58:29.864560 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:29.864568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:29.864651 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:29.864758 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:29.866238 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.867303 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.869305 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:29.870047 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:29.870195 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:29.875815 (+ 5620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:42859: BlockingRecv error: recv got EOF from 127.23.222.66:42859 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 15:58:30 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:30 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:30.339229 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.330227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:52387)
0929 15:58:30.330853 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:30.330871 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:30.331006 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:30.331623 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:30.331627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:30.332037 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:30.332233 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.332242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.333479 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.333484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.334420 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.334433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.335175 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.335179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.335390 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.336101 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:30.336121 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:30.338980 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":367,"threads_started":1}
W20260929 15:58:30.339586 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:30.330375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:52387 (local address 127.23.222.126:32943)
0929 15:58:30.330631 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:30.330638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:30.330959 (+ 321us) server_negotiation.cc:407] Connection header received
0929 15:58:30.331363 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:30.331368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:30.331446 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:30.331546 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:30.332425 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.333300 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.334584 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.335005 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.335109 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.339317 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:52387: BlockingRecv error: recv got EOF from 127.23.222.67:52387 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 15:58:30 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:30 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:30.413097 1244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.404322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:54129)
0929 15:58:30.404851 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:30.404867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:30.405013 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:30.405420 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:30.405424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:30.405828 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:30.405998 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.406007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.407100 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.407105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.408279 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.408293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.409008 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.409012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.409212 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.409834 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:30.409852 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:30.412837 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":281,"threads_started":1}
W20260929 15:58:30.413401 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:30.404411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:54129 (local address 127.23.222.126:32943)
0929 15:58:30.404630 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:30.404634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:30.404947 (+ 313us) server_negotiation.cc:407] Connection header received
0929 15:58:30.405158 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:30.405162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:30.405257 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:30.405358 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:30.406156 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.406909 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.408443 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.408867 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.408973 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.413086 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:54129: BlockingRecv error: recv got EOF from 127.23.222.65:54129 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 29 15:58:30 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:30 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:30.886483 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:30.877312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:33183)
0929 15:58:30.877878 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:30.877895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:30.878086 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:30.878452 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:30.878456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:30.878862 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:30.879032 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.879041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.880155 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.880161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.881213 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:30.881230 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.882259 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.882265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:30.882459 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.883156 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:30.883174 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:30.886116 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":315,"threads_started":1}
W20260929 15:58:30.886590 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:30.877457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:33183 (local address 127.23.222.126:32943)
0929 15:58:30.877721 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:30.877729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:30.877999 (+ 270us) server_negotiation.cc:407] Connection header received
0929 15:58:30.878227 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:30.878233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:30.878304 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:30.878420 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:30.879193 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.879991 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.881441 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:30.882096 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:30.882259 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:30.886342 (+ 4083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:33183: BlockingRecv error: recv got EOF from 127.23.222.66:33183 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 29 15:58:31 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:31 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:31.349345 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.340440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:53837)
0929 15:58:31.341034 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.341050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.341231 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.341591 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.341595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.341971 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:31.342155 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.342164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.343505 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.343512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.344483 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.344495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.345232 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.345238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.345416 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.346049 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:31.346067 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:31.348933 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":396,"threads_started":1}
W20260929 15:58:31.349412 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:31.340551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:53837 (local address 127.23.222.126:32943)
0929 15:58:31.340736 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.340741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.341151 (+ 410us) server_negotiation.cc:407] Connection header received
0929 15:58:31.341376 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.341382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.341454 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.341560 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:31.342335 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.343374 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.344630 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.345077 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.345209 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.349184 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:53837: BlockingRecv error: recv got EOF from 127.23.222.67:53837 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 15:58:31 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:31 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:31.423470 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.414136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:56539)
0929 15:58:31.414654 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.414671 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.414791 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.415128 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.415131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.415511 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:31.415686 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.415694 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.416767 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.416772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.418100 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.418115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.418974 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.418980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.419179 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.419984 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:31.420004 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:31.423173 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":265,"threads_started":1}
W20260929 15:58:31.423775 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:31.414291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:56539 (local address 127.23.222.126:32943)
0929 15:58:31.414554 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.414560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.414757 (+ 197us) server_negotiation.cc:407] Connection header received
0929 15:58:31.414895 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.414899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.414971 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.415071 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:31.415811 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.416606 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.418269 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.418792 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.418944 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.423455 (+ 4511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:56539: BlockingRecv error: recv got EOF from 127.23.222.65:56539 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 29 15:58:31 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:31 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:31.896385 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:31.887623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:60585)
0929 15:58:31.888186 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:31.888208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:31.888361 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:31.888740 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:31.888746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:31.889154 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:31.889306 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.889315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.890292 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.890299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.891317 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:31.891330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.892057 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.892063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:31.892245 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.892855 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:31.892872 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:31.896106 (+ 3234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":272,"threads_started":1}
W20260929 15:58:31.896653 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:31.887709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:60585 (local address 127.23.222.126:32943)
0929 15:58:31.887956 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:31.887964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:31.888285 (+ 321us) server_negotiation.cc:407] Connection header received
0929 15:58:31.888505 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:31.888509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:31.888570 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:31.888677 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:31.889457 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.890147 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.891466 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:31.891895 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:31.892036 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:31.896381 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:60585: BlockingRecv error: recv got EOF from 127.23.222.66:60585 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 15:58:32 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:32 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:32.359145 1249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.350414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:50079)
0929 15:58:32.350979 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:32.351003 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:32.351156 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:32.351564 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:32.351567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:32.351936 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:32.352089 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.352097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.353236 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.353241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.354221 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.354235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.355005 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.355010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.355147 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.355762 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:32.355779 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:32.358847 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":272,"threads_started":1}
W20260929 15:58:32.359351 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:32.350487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:50079 (local address 127.23.222.126:32943)
0929 15:58:32.350763 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.350770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.351100 (+ 330us) server_negotiation.cc:407] Connection header received
0929 15:58:32.351325 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.351329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.351392 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.351476 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:32.352252 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.353058 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.354375 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.354806 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.354905 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.359083 (+ 4178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:50079: BlockingRecv error: recv got EOF from 127.23.222.67:50079 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 29 15:58:32 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:32 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:32.433133 1250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.424482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:49387)
0929 15:58:32.424961 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:32.424977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:32.425104 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:32.425456 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:32.425459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:32.425851 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:32.426030 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.426058 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.427153 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.427158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.428187 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.428200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.428936 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.428940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.429085 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.429698 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:32.429715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:32.432805 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":236,"threads_started":1}
W20260929 15:58:32.433362 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:32.424543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:49387 (local address 127.23.222.126:32943)
0929 15:58:32.424788 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.424794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.425075 (+ 281us) server_negotiation.cc:407] Connection header received
0929 15:58:32.425206 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.425210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.425284 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.425364 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:32.426241 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.427001 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.428334 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.428775 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.428877 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.433071 (+ 4194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:49387: BlockingRecv error: recv got EOF from 127.23.222.65:49387 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 15:58:32 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:32 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:32.907464 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:32.897826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:47989)
0929 15:58:32.898586 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:32.898603 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:32.898719 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:32.899078 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:32.899081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:32.899718 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:32.899884 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.899893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.901004 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.901008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.902088 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:32.902103 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.903107 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.903119 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:32.903344 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.904005 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:32.904023 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:32.907067 (+ 3044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":455,"threads_started":1}
W20260929 15:58:32.907778 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:32.898150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:47989 (local address 127.23.222.126:32943)
0929 15:58:32.898369 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:32.898376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:32.898705 (+ 329us) server_negotiation.cc:407] Connection header received
0929 15:58:32.898817 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:32.898824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:32.898904 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:32.898997 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:32.900029 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.900856 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.902249 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:32.902939 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:32.903050 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:32.907407 (+ 4357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:47989: BlockingRecv error: recv got EOF from 127.23.222.66:47989 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 15:58:33 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:33 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:33.369053 1252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.360298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:57183)
0929 15:58:33.360903 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:33.360920 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:33.361037 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:33.361389 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:33.361392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:33.361807 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:33.361976 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.361984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.363100 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.363104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.364152 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.364164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.364918 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.364923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.365068 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.365808 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:33.365828 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:33.368692 (+ 2864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":347,"threads_started":1}
W20260929 15:58:33.369236 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:33.360486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:57183 (local address 127.23.222.126:32943)
0929 15:58:33.360682 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.360687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.361022 (+ 335us) server_negotiation.cc:407] Connection header received
0929 15:58:33.361159 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.361163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.361240 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.361336 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:33.362131 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.362931 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.364306 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.364753 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.364858 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.368956 (+ 4098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:57183: BlockingRecv error: recv got EOF from 127.23.222.67:57183 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 15:58:33 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:33 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:33.442405 1253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.434271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:47703)
0929 15:58:33.434781 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:33.434796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:33.434951 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:33.435336 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:33.435340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:33.435752 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:33.435905 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.435913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.436920 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.436928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.437826 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.437837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.438539 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.438545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.438725 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.439384 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:33.439402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:33.442071 (+ 2669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":285,"threads_started":1}
W20260929 15:58:33.442543 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:33.434348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:47703 (local address 127.23.222.126:32943)
0929 15:58:33.434583 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.434592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.434887 (+ 295us) server_negotiation.cc:407] Connection header received
0929 15:58:33.435077 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.435089 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.435171 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.435308 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:33.436045 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.436782 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.437966 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.438382 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.438517 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.442274 (+ 3757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:47703: BlockingRecv error: recv got EOF from 127.23.222.65:47703 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 15:58:33 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:33 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:33.916815 1254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:33.908491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:47177)
0929 15:58:33.909074 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:33.909090 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:33.909207 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:33.909565 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:33.909572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:33.909982 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:33.910141 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.910150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.911175 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.911179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.912111 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:33.912122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.912798 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.912802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:33.912923 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.913579 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:33.913603 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:33.916501 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":344,"threads_started":1}
W20260929 15:58:33.917054 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:33.908606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:47177 (local address 127.23.222.126:32943)
0929 15:58:33.908791 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:33.908796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:33.909163 (+ 367us) server_negotiation.cc:407] Connection header received
0929 15:58:33.909303 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:33.909307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:33.909370 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:33.909456 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:33.910265 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.911032 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.912242 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:33.912656 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:33.912757 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:33.916811 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:47177: BlockingRecv error: recv got EOF from 127.23.222.66:47177 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 15:58:34 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:34 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:34.387897 1255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:34.370116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:40239)
0929 15:58:34.370641 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:34.370659 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:34.370784 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:34.371097 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:34.371100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:34.371610 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:34.371773 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:34.371783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.372955 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.372977 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:34.373977 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:34.373990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.374804 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.374809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:34.374960 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:34.384387 (+ 9427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:34.384408 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:34.387671 (+ 3263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":282,"threads_started":1}
W20260929 15:58:34.388221 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:34.370335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:40239 (local address 127.23.222.126:32943)
0929 15:58:34.370588 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:34.370594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:34.370735 (+ 141us) server_negotiation.cc:407] Connection header received
0929 15:58:34.370890 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:34.370894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:34.370956 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:34.371040 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:34.371952 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.372791 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.374157 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.374615 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.374722 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:34.387916 (+ 13194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:40239: BlockingRecv error: recv got EOF from 127.23.222.67:40239 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 15:58:34 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:34 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:34.452086 1256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:34.443597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:43459)
0929 15:58:34.444269 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:34.444285 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:34.444394 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:34.444744 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:34.444747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:34.445123 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:34.445287 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:34.445295 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.446279 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.446282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:34.447200 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:34.447211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.447927 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.447931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:34.448043 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:34.448606 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:34.448624 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:34.451712 (+ 3088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":383,"threads_started":1}
W20260929 15:58:34.452176 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:34.443769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:43459 (local address 127.23.222.126:32943)
0929 15:58:34.443989 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:34.443999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:34.444366 (+ 367us) server_negotiation.cc:407] Connection header received
0929 15:58:34.444489 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:34.444493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:34.444588 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:34.444670 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:34.445412 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.446157 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.447379 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.447793 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.447892 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:34.451922 (+ 4030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:43459: BlockingRecv error: recv got EOF from 127.23.222.65:43459 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 15:58:34 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:34 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:34.926770 1257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:34.918156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:55677)
0929 15:58:34.918755 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:34.918771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:34.918952 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:34.919413 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:34.919418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:34.919930 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:34.920099 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:34.920109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.921184 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.921188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:34.922216 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:34.922229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.922873 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.922878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:34.922989 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:34.923665 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:34.923683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:34.926516 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":306,"threads_started":1}
W20260929 15:58:34.926988 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:34.918191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:55677 (local address 127.23.222.126:32943)
0929 15:58:34.918444 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:34.918451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:34.918877 (+ 426us) server_negotiation.cc:407] Connection header received
0929 15:58:34.919148 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:34.919153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:34.919233 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:34.919363 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:34.920237 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.921037 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.922350 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:34.922734 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:34.922828 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:34.926741 (+ 3913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:55677: BlockingRecv error: recv got EOF from 127.23.222.66:55677 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 29 15:58:35 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:35 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:35.398491 1258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:35.389048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:54139)
0929 15:58:35.389868 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:35.389884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:35.390063 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:35.390529 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:35.390534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:35.391159 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:35.391439 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:35.391451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.392518 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.392526 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:35.393535 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:35.393548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.394359 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.394365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:35.394566 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:35.395153 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:35.395172 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:35.398240 (+ 3068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":506,"threads_started":1}
W20260929 15:58:35.398664 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:35.389161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:54139 (local address 127.23.222.126:32943)
0929 15:58:35.389501 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:35.389511 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:35.390019 (+ 508us) server_negotiation.cc:407] Connection header received
0929 15:58:35.390229 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:35.390238 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:35.390327 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:35.390441 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:35.391593 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.392353 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.393699 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.394187 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.394342 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:35.398458 (+ 4116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:54139: BlockingRecv error: recv got EOF from 127.23.222.67:54139 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 29 15:58:35 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:35 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:35.462695 1259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:35.453164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:43897)
0929 15:58:35.453779 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:35.453797 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:35.453916 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:35.454229 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:35.454233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:35.454669 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:35.454846 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:35.454855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.456102 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.456115 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:35.457139 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:35.457153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.458061 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.458071 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:35.458254 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:35.458902 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:35.458921 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:35.462427 (+ 3506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":346,"threads_started":1}
W20260929 15:58:35.463157 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:35.453328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:43897 (local address 127.23.222.126:32943)
0929 15:58:35.453499 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:35.453505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:35.453875 (+ 370us) server_negotiation.cc:407] Connection header received
0929 15:58:35.454011 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:35.454014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:35.454089 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:35.454173 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:35.455020 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.455923 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.457313 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.457825 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.457977 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:35.462684 (+ 4707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:43897: BlockingRecv error: recv got EOF from 127.23.222.65:43897 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 15:58:35 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:35 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:35.936882 1260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:35.927906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:57779)
0929 15:58:35.928455 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:35.928473 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:35.928604 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:35.928976 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:35.928979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:35.929382 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:35.929542 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:35.929552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.930925 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.930930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:35.931939 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:35.931953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.932704 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.932708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:35.932823 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:35.933486 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:35.933505 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:35.936623 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":305,"threads_started":1}
W20260929 15:58:35.937069 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:35.928072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:57779 (local address 127.23.222.126:32943)
0929 15:58:35.928314 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:35.928320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:35.928559 (+ 239us) server_negotiation.cc:407] Connection header received
0929 15:58:35.928710 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:35.928714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:35.928779 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:35.928866 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:35.929734 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.930767 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.932129 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:35.932544 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:35.932642 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:35.936837 (+ 4195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:57779: BlockingRecv error: recv got EOF from 127.23.222.66:57779 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 29 15:58:36 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:36 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:36.408936 1261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:36.399791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:52321)
0929 15:58:36.400414 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:36.400431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:36.400629 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:36.401077 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:36.401082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:36.401516 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:36.401695 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:36.401704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.402783 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.402787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:36.403853 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:36.403870 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.404872 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.404876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:36.405045 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:36.405805 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:36.405823 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:36.408697 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":332,"threads_started":1}
W20260929 15:58:36.409219 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:36.399920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:52321 (local address 127.23.222.126:32943)
0929 15:58:36.400194 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:36.400202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:36.400628 (+ 426us) server_negotiation.cc:407] Connection header received
0929 15:58:36.400787 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:36.400792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:36.400860 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:36.400958 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:36.401885 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.402647 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.404005 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.404683 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.404805 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:36.408903 (+ 4098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:52321: BlockingRecv error: recv got EOF from 127.23.222.67:52321 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 15:58:36 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:36 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:36.473042 1262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:36.463799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:55153)
0929 15:58:36.464384 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:36.464401 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:36.464608 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:36.465029 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:36.465034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:36.465501 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:36.465678 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:36.465687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.466827 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.466834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:36.467948 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:36.467962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.468735 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.468741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:36.468942 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:36.469619 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:36.469639 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:36.472738 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":301,"threads_started":1}
W20260929 15:58:36.473294 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:36.463895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:55153 (local address 127.23.222.126:32943)
0929 15:58:36.464183 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:36.464191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:36.464537 (+ 346us) server_negotiation.cc:407] Connection header received
0929 15:58:36.464763 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:36.464776 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:36.464844 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:36.464987 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:36.465855 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.466660 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.468107 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.468561 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.468726 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:36.473012 (+ 4286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:55153: BlockingRecv error: recv got EOF from 127.23.222.65:55153 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 29 15:58:36 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:36 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:36.947103 1263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:36.937955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:45555)
0929 15:58:36.938605 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:36.938622 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:36.938800 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:36.939188 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:36.939193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:36.939758 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:36.939948 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:36.939959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.941055 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.941062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:36.942349 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:36.942363 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.943346 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.943355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:36.943501 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:36.944110 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:36.944128 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:36.946852 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":328,"threads_started":1}
W20260929 15:58:36.947355 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:36.938117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:45555 (local address 127.23.222.126:32943)
0929 15:58:36.938368 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:36.938378 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:36.938726 (+ 348us) server_negotiation.cc:407] Connection header received
0929 15:58:36.938970 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:36.938975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:36.939042 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:36.939155 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:36.940112 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.940871 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.942505 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:36.943148 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:36.943326 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:36.947096 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:45555: BlockingRecv error: recv got EOF from 127.23.222.66:45555 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 29 15:58:37 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:37 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:37.418753 1264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:37.410081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:49663)
0929 15:58:37.410624 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:37.410641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:37.410778 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:37.411181 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:37.411186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:37.411638 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:37.411809 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:37.411819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.412885 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.412889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:37.413897 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:37.413918 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.414723 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.414728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:37.414854 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:37.415486 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:37.415504 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:37.418394 (+ 2890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":292,"threads_started":1}
W20260929 15:58:37.418831 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:37.410098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:49663 (local address 127.23.222.126:32943)
0929 15:58:37.410298 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:37.410305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:37.410727 (+ 422us) server_negotiation.cc:407] Connection header received
0929 15:58:37.410904 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:37.410908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:37.411027 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:37.411122 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:37.411978 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.412720 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.414071 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.414513 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.414683 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:37.418601 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:49663: BlockingRecv error: recv got EOF from 127.23.222.67:49663 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 15:58:37 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:37 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:37.482928 1265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:37.474156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:49335)
0929 15:58:37.474794 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:37.474812 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:37.474933 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:37.475285 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:37.475289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:37.475712 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:37.475872 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:37.475880 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.477001 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.477008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:37.478196 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:37.478211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.479002 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.479008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:37.479136 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:37.479824 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:37.479843 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:37.482681 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":375,"threads_started":1}
W20260929 15:58:37.483140 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:37.474189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:49335 (local address 127.23.222.126:32943)
0929 15:58:37.474358 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:37.474363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:37.474915 (+ 552us) server_negotiation.cc:407] Connection header received
0929 15:58:37.475033 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:37.475037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:37.475120 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:37.475203 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:37.476036 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.476799 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.478347 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.478804 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.478903 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:37.482874 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:49335: BlockingRecv error: recv got EOF from 127.23.222.65:49335 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 15:58:37 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:37 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:37.956341 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:37.948060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:49773)
0929 15:58:37.948560 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:37.948578 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:37.948702 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:37.949006 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:37.949010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:37.949428 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:37.949619 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:37.949629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.950757 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.950762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:37.951837 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:37.951850 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.952543 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.952548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:37.952662 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:37.953343 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:37.953361 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:37.956119 (+ 2758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":272,"threads_started":1}
W20260929 15:58:37.956594 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:37.948218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:49773 (local address 127.23.222.126:32943)
0929 15:58:37.948443 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:37.948449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:37.948663 (+ 214us) server_negotiation.cc:407] Connection header received
0929 15:58:37.948804 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:37.948808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:37.948868 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:37.948949 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:37.949777 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.950599 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.951964 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:37.952364 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:37.952510 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:37.956349 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:49773: BlockingRecv error: recv got EOF from 127.23.222.66:49773 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 15:58:38 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:38 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:38.428503 1267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:38.419866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:40981)
0929 15:58:38.420427 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:38.420445 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:38.420570 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:38.420986 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:38.420993 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:38.421411 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:38.421580 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:38.421589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.422643 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.422647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:38.423667 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:38.423680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.424422 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.424427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:38.424578 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:38.425250 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:38.425269 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:38.428161 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":327,"threads_started":1}
W20260929 15:58:38.428678 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:38.420089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:40981 (local address 127.23.222.126:32943)
0929 15:58:38.420344 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:38.420350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:38.420548 (+ 198us) server_negotiation.cc:407] Connection header received
0929 15:58:38.420703 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:38.420708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:38.420780 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:38.420871 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:38.421744 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.422508 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.423813 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.424248 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.424353 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:38.428380 (+ 4027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:40981: BlockingRecv error: recv got EOF from 127.23.222.67:40981 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Sep 29 15:58:38 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:38 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:38.492671 1268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:38.483870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:54267)
0929 15:58:38.484424 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:38.484441 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:38.484551 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:38.484831 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:38.484834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:38.485236 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:38.485418 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:38.485427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.486413 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.486417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:38.487461 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:38.487475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.488203 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.488208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:38.488331 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:38.488947 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:38.488964 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:38.492332 (+ 3368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":317,"threads_started":1}
W20260929 15:58:38.492751 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:38.483972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:54267 (local address 127.23.222.126:32943)
0929 15:58:38.484155 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:38.484160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:38.484512 (+ 352us) server_negotiation.cc:407] Connection header received
0929 15:58:38.484647 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:38.484651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:38.484709 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:38.484793 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:38.485536 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.486253 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.487609 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.488060 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.488167 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:38.492567 (+ 4400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:54267: BlockingRecv error: recv got EOF from 127.23.222.65:54267 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 15:58:38 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:38 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:38.966297 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:38.957252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:53275)
0929 15:58:38.957789 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:38.957805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:38.957957 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:38.958335 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:38.958339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:38.958820 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:38.958993 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:38.959003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.960200 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.960206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:38.961143 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:38.961156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.961938 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.961944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:38.962143 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:38.962823 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:38.962842 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:38.965924 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":274,"threads_started":1}
W20260929 15:58:38.966403 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:38.957430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:53275 (local address 127.23.222.126:32943)
0929 15:58:38.957652 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:38.957658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:38.957886 (+ 228us) server_negotiation.cc:407] Connection header received
0929 15:58:38.958111 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:38.958116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:38.958203 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:38.958308 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:38.959155 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.960056 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.961290 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:38.961776 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:38.961918 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:38.966157 (+ 4239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:53275: BlockingRecv error: recv got EOF from 127.23.222.66:53275 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 29 15:58:39 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:39 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:39.443753 1270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:39.429573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:49347)
0929 15:58:39.430119 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:39.430136 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:39.430254 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:39.430662 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:39.430666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:39.431093 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:39.431307 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.431317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.432530 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.432536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.433741 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.433754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.434498 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.434504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.434686 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.440384 (+ 5698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:39.440408 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:39.443381 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":300,"threads_started":1}
W20260929 15:58:39.443881 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:39.429756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:49347 (local address 127.23.222.126:32943)
0929 15:58:39.429942 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:39.429947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:39.430220 (+ 273us) server_negotiation.cc:407] Connection header received
0929 15:58:39.430418 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:39.430422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:39.430487 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:39.430582 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:39.431461 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.432384 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.433884 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.434332 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.434479 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.443639 (+ 9160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:49347: BlockingRecv error: recv got EOF from 127.23.222.67:49347 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 15:58:39 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:39 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:39.502931 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:39.493853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:45605)
0929 15:58:39.494439 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:39.494460 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:39.494669 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:39.495024 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:39.495028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:39.495584 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:39.495786 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.495795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.496860 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.496866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.497827 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.497841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.498693 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.498699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.498904 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.499643 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:39.499661 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:39.502691 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":322,"threads_started":1}
W20260929 15:58:39.503379 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:39.493986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:45605 (local address 127.23.222.126:32943)
0929 15:58:39.494234 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:39.494242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:39.494574 (+ 332us) server_negotiation.cc:407] Connection header received
0929 15:58:39.494808 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:39.494814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:39.494885 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:39.495002 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:39.495927 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.496709 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.497981 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.498529 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.498675 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.502949 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:45605: BlockingRecv error: recv got EOF from 127.23.222.65:45605 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 15:58:39 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:39 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:39.977046 1272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:39.967663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:55463)
0929 15:58:39.968255 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:39.968272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:39.968390 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:39.968766 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:39.968770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:39.969195 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:39.969354 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.969362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.970395 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.970399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.971440 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:39.971454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.972373 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.972378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:39.972522 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.973176 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:39.973196 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:39.976769 (+ 3573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":363,"threads_started":1}
W20260929 15:58:39.977319 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:39.967828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:55463 (local address 127.23.222.126:32943)
0929 15:58:39.968002 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:39.968007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:39.968351 (+ 344us) server_negotiation.cc:407] Connection header received
0929 15:58:39.968498 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:39.968502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:39.968570 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:39.968668 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:39.969518 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.970235 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.971646 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:39.972202 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:39.972311 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:39.977065 (+ 4754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:55463: BlockingRecv error: recv got EOF from 127.23.222.66:55463 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 15:58:40 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:40 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:40.453536 1273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:40.445019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:35855)
0929 15:58:40.445477 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:40.445495 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:40.445610 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:40.445991 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:40.445996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:40.446445 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:40.446626 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:40.446636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.447729 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.447733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:40.448653 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:40.448665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.449406 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.449410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:40.449547 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.450167 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:40.450184 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:40.453287 (+ 3103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":269,"threads_started":1}
W20260929 15:58:40.453783 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:40.445092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:35855 (local address 127.23.222.126:32943)
0929 15:58:40.445334 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:40.445340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:40.445578 (+ 238us) server_negotiation.cc:407] Connection header received
0929 15:58:40.445714 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:40.445717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:40.445780 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:40.445862 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:40.446781 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.447593 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.448820 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.449262 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.449364 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.453538 (+ 4174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:35855: BlockingRecv error: recv got EOF from 127.23.222.67:35855 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 15:58:40 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:40 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:40.513936 1274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:40.504048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:51303)
0929 15:58:40.504715 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:40.504731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:40.504905 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:40.505290 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:40.505294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:40.505712 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:40.505869 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:40.505877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.507006 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.507013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:40.507997 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:40.508009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.508899 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.508906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:40.509192 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.509898 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:40.509939 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:40.513336 (+ 3397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":337,"threads_started":1}
W20260929 15:58:40.514122 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:40.504224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:51303 (local address 127.23.222.126:32943)
0929 15:58:40.504530 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:40.504538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:40.504831 (+ 293us) server_negotiation.cc:407] Connection header received
0929 15:58:40.505062 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:40.505073 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:40.505140 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:40.505252 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:40.506034 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.506851 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.508159 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.508635 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.508798 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.513888 (+ 5090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:51303: BlockingRecv error: recv got EOF from 127.23.222.65:51303 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 15:58:40 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:40 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:40.986470 1275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:40.978442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:35347)
0929 15:58:40.979031 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:40.979048 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:40.979160 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:40.979530 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:40.979534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:40.979895 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:40.980065 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:40.980073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.981081 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.981084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:40.981994 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:40.982013 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.982699 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.982703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:40.982814 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.983518 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:40.983535 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:40.986226 (+ 2691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":313,"threads_started":1}
W20260929 15:58:40.986769 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:40.978654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:35347 (local address 127.23.222.126:32943)
0929 15:58:40.978854 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:40.978859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:40.979167 (+ 308us) server_negotiation.cc:407] Connection header received
0929 15:58:40.979323 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:40.979327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:40.979400 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:40.979488 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:40.980198 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.980966 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.982148 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:40.982549 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:40.982646 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:40.986444 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:35347: BlockingRecv error: recv got EOF from 127.23.222.66:35347 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 29 15:58:41 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:41 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:41.463040 1276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:41.454630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:48207)
0929 15:58:41.455204 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:41.455220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:41.455404 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:41.455759 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:41.455762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:41.456289 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:41.456450 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:41.456459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.457483 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.457488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:41.458419 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:41.458431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.459151 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.459157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:41.459342 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:41.459956 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:41.459974 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:41.462690 (+ 2716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":309,"threads_started":1}
W20260929 15:58:41.463156 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:41.454699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:48207 (local address 127.23.222.126:32943)
0929 15:58:41.454889 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:41.454894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:41.455357 (+ 463us) server_negotiation.cc:407] Connection header received
0929 15:58:41.455518 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:41.455522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:41.455585 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:41.455679 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:41.456601 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.457341 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.458569 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.458972 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.459113 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:41.462889 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:48207: BlockingRecv error: recv got EOF from 127.23.222.67:48207 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 15:58:41 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:41 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:41.524921 1277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:41.515502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:39081)
0929 15:58:41.516046 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:41.516065 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:41.516219 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:41.516606 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:41.516609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:41.517085 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:41.517290 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:41.517299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.518647 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.518652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:41.519714 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:41.519728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.520545 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.520550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:41.520727 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:41.521432 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:41.521451 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:41.524525 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":418,"threads_started":1}
W20260929 15:58:41.525005 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:41.515663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:39081 (local address 127.23.222.126:32943)
0929 15:58:41.515938 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:41.515945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:41.516164 (+ 219us) server_negotiation.cc:407] Connection header received
0929 15:58:41.516338 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:41.516343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:41.516428 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:41.516562 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:41.517445 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.518499 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.519866 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.520381 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.520515 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:41.524745 (+ 4230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:39081: BlockingRecv error: recv got EOF from 127.23.222.65:39081 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 29 15:58:41 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:41 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:41.995931 1278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:41.987446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:55643)
0929 15:58:41.987993 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:41.988010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:41.988126 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:41.988507 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:41.988512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:41.989014 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:41.989220 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:41.989230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.990502 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.990506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:41.991436 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:41.991449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.992088 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.992092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:41.992206 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:41.992778 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:41.992794 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:41.995677 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":287,"threads_started":1}
W20260929 15:58:41.996098 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:41.987574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:55643 (local address 127.23.222.126:32943)
0929 15:58:41.987770 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:41.987775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:41.988097 (+ 322us) server_negotiation.cc:407] Connection header received
0929 15:58:41.988245 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:41.988250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:41.988337 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:41.988426 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:41.989411 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.990364 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.991568 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:41.991960 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:41.992064 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:41.995881 (+ 3817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:55643: BlockingRecv error: recv got EOF from 127.23.222.66:55643 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 15:58:42 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.67@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:42.472206 1279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:42.463998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.67:41651)
0929 15:58:42.464474 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:42.464498 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:42.464617 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:42.464917 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:42.464920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:42.465282 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:42.465453 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.465461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.466458 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.466461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.467407 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.467418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.468085 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.468089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.468213 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.468883 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:42.468901 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:42.471896 (+ 2995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":263,"threads_started":1}
W20260929 15:58:42.472493 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:42.464185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:41651 (local address 127.23.222.126:32943)
0929 15:58:42.464453 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:42.464460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:42.464578 (+ 118us) server_negotiation.cc:407] Connection header received
0929 15:58:42.464724 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:42.464728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:42.464788 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:42.464868 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:42.465596 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.466332 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.467522 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.467933 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.468036 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.472218 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.67:41651: BlockingRecv error: recv got EOF from 127.23.222.67:41651 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 15:58:42 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:42 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.65@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:42.534821 1280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:42.526073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.65:36043)
0929 15:58:42.526566 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:42.526582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:42.526692 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:42.527027 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:42.527030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:42.527626 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:42.527810 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.527819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.528824 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.528828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.529767 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.529778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.530442 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.530446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:42.530571 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.531144 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:42.531162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:42.534561 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":268,"threads_started":1}
W20260929 15:58:42.534916 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:42.526228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:36043 (local address 127.23.222.126:32943)
0929 15:58:42.526467 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:42.526473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:42.526665 (+ 192us) server_negotiation.cc:407] Connection header received
0929 15:58:42.526791 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:42.526795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:42.526854 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:42.526930 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:42.527938 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.528683 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.529887 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.530307 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.530396 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:42.534739 (+ 4343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.65:36043: BlockingRecv error: recv got EOF from 127.23.222.65:36043 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 15:58:43 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 15:58:43 dist-test-slave-cq88 krb5kdc[32694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.222.66@KRBTEST.COM for oryx/127.23.222.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 15:58:43.005817 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:42.997104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:32943 (local address 127.23.222.66:60813)
0929 15:58:42.997680 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:42.997697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:42.997809 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:42.998136 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:42.998139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:42.998541 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:42.998709 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:42.998719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.999851 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:42.999855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.000883 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:43.000895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.001697 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.001702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:43.001882 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.002567 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:43.002585 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:43.005455 (+ 2870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.222.126:32943: Server oryx/127.23.222.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":317,"threads_started":1}
W20260929 15:58:43.005934 970 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:42.997264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:60813 (local address 127.23.222.126:32943)
0929 15:58:42.997492 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:42.997497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:42.997808 (+ 311us) server_negotiation.cc:407] Connection header received
0929 15:58:42.997912 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:42.997916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:42.997987 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:42.998088 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:42.998863 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:42.999710 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.001036 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.001547 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.001683 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.005638 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.222.66:60813: BlockingRecv error: recv got EOF from 127.23.222.66:60813 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:58:43.286788 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 819
I20260929 15:58:43.299494 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:43.288067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.23.222.65:45591)
0929 15:58:43.288601 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:43.288608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:43.288635 (+ 27us) server_negotiation.cc:407] Connection header received
0929 15:58:43.288736 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:43.288742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:43.288973 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:43.289138 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:43.290586 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.291652 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.292930 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.293525 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.293627 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.296295 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:43.296304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:43.296317 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:43.296362 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:43.298822 (+ 2460us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.222.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:58:43.299087 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44244: Request ticket server kudu/127.23.222.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":284,"threads_started":1}
W20260929 15:58:43.299606 1283 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44244: Request ticket server kudu/127.23.222.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:58:43.299628 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 819: Not authorized: Client connection negotiation failed: client connection to 127.23.222.65:45591: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.222.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:58:43.299845 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 819
I20260929 15:58:43.317302 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 972
I20260929 15:58:43.330358 1284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:43.318368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.23.222.66:43209)
0929 15:58:43.318977 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:43.318985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:43.319016 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:58:43.319082 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:43.319089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:43.319367 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:43.319539 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:43.321435 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.322618 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.323992 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.324568 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.324669 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.327196 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:43.327205 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:43.327219 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:43.327305 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:43.329864 (+ 2559us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.222.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:58:43.330129 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37404: Request ticket server kudu/127.23.222.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":298,"threads_started":1}
W20260929 15:58:43.330487 1284 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37404: Request ticket server kudu/127.23.222.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:58:43.330611 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 972: Not authorized: Client connection negotiation failed: client connection to 127.23.222.66:43209: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.222.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:58:43.330788 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 972
I20260929 15:58:43.348078 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 1103
I20260929 15:58:43.360903 1285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:43.349233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.23.222.67:36753)
0929 15:58:43.349808 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:43.349815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:43.349841 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:58:43.349900 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:43.349906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:43.350181 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:43.350332 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:43.351745 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.352765 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.354056 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:43.354619 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:43.354721 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:43.357782 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:43.357792 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:43.357806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:43.357843 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:43.360332 (+ 2489us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.222.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 15:58:43.360628 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46924: Request ticket server kudu/127.23.222.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":312,"threads_started":1}
W20260929 15:58:43.361011 1285 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46924: Request ticket server kudu/127.23.222.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 15:58:43.361088 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 1103: Not authorized: Client connection negotiation failed: client connection to 127.23.222.67:36753: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.222.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 15:58:43.361286 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 1103
I20260929 15:58:43.378213 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 750
I20260929 15:58:43.379640 812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:43.497213 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 750
2026-09-29T15:58:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21282 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T15:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:43Z Disabled control of system clock
I20260929 15:58:43.576500 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38387
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:38387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 15:58:43.918558 1290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:43.918871 1290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:43.918985 1290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:43.926580 1290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:43.926734 1290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:43.926810 1290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:43.926921 1290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:43.927014 1290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:43.936798 1290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:38387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:43.938359 1290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:43.940251 1290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:43.953787 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:43.953724 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:43.953827 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:43.954128 1290 server_base.cc:1034] running on GCE node
I20260929 15:58:43.955116 1290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:43.956616 1290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:43.957839 1290 hybrid_clock.cc:648] HybridClock initialized: now 1790697523957774 us; error 46 us; skew 500 ppm
I20260929 15:58:43.958240 1290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:43.961571 1290 webserver.cc:460] Webserver started at http://127.23.222.126:37651/ using document root <none> and password file <none>
I20260929 15:58:43.962563 1290 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:43.962690 1290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:43.963099 1290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:43.966910 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "30f98d17adbd470ca19a8c8dc5a3d325"
format_stamp: "Formatted at 2026-09-29 15:58:43 on dist-test-slave-cq88"
server_key: "453952edef9e84d0dad2e2d748add32c"
server_key_iv: "b84a1fc77c7795f734a77867d19faed2"
server_key_version: "encryptionkey@0"
I20260929 15:58:43.967724 1290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "30f98d17adbd470ca19a8c8dc5a3d325"
format_stamp: "Formatted at 2026-09-29 15:58:43 on dist-test-slave-cq88"
server_key: "453952edef9e84d0dad2e2d748add32c"
server_key_iv: "b84a1fc77c7795f734a77867d19faed2"
server_key_version: "encryptionkey@0"
I20260929 15:58:43.972957 1290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 15:58:43.976594 1305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:43.977988 1290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:43.978178 1290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "30f98d17adbd470ca19a8c8dc5a3d325"
format_stamp: "Formatted at 2026-09-29 15:58:43 on dist-test-slave-cq88"
server_key: "453952edef9e84d0dad2e2d748add32c"
server_key_iv: "b84a1fc77c7795f734a77867d19faed2"
server_key_version: "encryptionkey@0"
I20260929 15:58:43.978435 1290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:44.002894 1290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:44.003950 1290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:44.004278 1290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:44.024823 1290 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:38387
I20260929 15:58:44.024843 1356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:38387 every 8 connection(s)
I20260929 15:58:44.026332 1290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:44.030189 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 1290
I20260929 15:58:44.030401 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:58:44.030745 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:44.031275 24441 external_mini_cluster.cc:1428] Setting key 6f1378c7c5b4aefaf0f8c8fd6287f906
I20260929 15:58:44.045104 1357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325: Bootstrap starting.
I20260929 15:58:44.046096 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:44.033540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.23.222.126:38387)
0929 15:58:44.034586 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:44.034608 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:44.034652 (+ 44us) server_negotiation.cc:407] Connection header received
0929 15:58:44.036962 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:44.036990 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:44.037525 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:44.037946 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:44.038634 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.040043 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.041415 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.042145 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.042371 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:44.043697 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:44.043704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:44.043719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:44.044274 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:44.044351 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:44.044373 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:44.044516 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:44.044737 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 15:58:44.045058 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":352,"threads_started":1}
I20260929 15:58:44.048830 1357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:44.049937 1357 log.cc:824] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:44.052950 1357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325: No bootstrap required, opened a new log
I20260929 15:58:44.060460 1357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f98d17adbd470ca19a8c8dc5a3d325" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38387 } }
I20260929 15:58:44.060861 1357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:44.060971 1357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f98d17adbd470ca19a8c8dc5a3d325, State: Initialized, Role: FOLLOWER
I20260929 15:58:44.061684 1357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f98d17adbd470ca19a8c8dc5a3d325" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38387 } }
I20260929 15:58:44.061980 1357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:44.062140 1357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:44.062327 1357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:44.064628 1357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f98d17adbd470ca19a8c8dc5a3d325" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38387 } }
I20260929 15:58:44.065161 1357 leader_election.cc:302] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f98d17adbd470ca19a8c8dc5a3d325; no voters:
I20260929 15:58:44.065891 1357 leader_election.cc:288] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:44.066051 1362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:44.067087 1362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [term 1 LEADER]: Becoming Leader. State: Replica: 30f98d17adbd470ca19a8c8dc5a3d325, State: Running, Role: LEADER
I20260929 15:58:44.067742 1362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f98d17adbd470ca19a8c8dc5a3d325" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38387 } }
I20260929 15:58:44.068506 1357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:44.070835 1364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f98d17adbd470ca19a8c8dc5a3d325. Latest consensus state: current_term: 1 leader_uuid: "30f98d17adbd470ca19a8c8dc5a3d325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f98d17adbd470ca19a8c8dc5a3d325" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38387 } } }
I20260929 15:58:44.071218 1364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:44.070990 1363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f98d17adbd470ca19a8c8dc5a3d325" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f98d17adbd470ca19a8c8dc5a3d325" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38387 } } }
I20260929 15:58:44.071427 1363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:44.073348 1369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:44.079407 1369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:44.088593 1369 catalog_manager.cc:1347] Generated new cluster ID: 51dc53c796934c3f834d669cb4d7de74
I20260929 15:58:44.088723 1369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:44.101410 1369 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:44.103449 1369 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:44.119122 1369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f98d17adbd470ca19a8c8dc5a3d325: Generated new TSK 0
I20260929 15:58:44.120379 1369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:58:44.129962 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38387
--builtin_ntp_servers=127.23.222.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 15:58:44.472034 1381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:44.472317 1381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:44.472406 1381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:44.479965 1381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:44.480064 1381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:44.480167 1381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:58:44.489516 1381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:44.490911 1381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:44.492841 1381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:44.506739 1386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:44.506676 1389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:44.506759 1387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:44.508067 1381 server_base.cc:1034] running on GCE node
I20260929 15:58:44.508778 1381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:44.509728 1381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:44.510938 1381 hybrid_clock.cc:648] HybridClock initialized: now 1790697524510893 us; error 35 us; skew 500 ppm
I20260929 15:58:44.511293 1381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:44.519121 1381 webserver.cc:460] Webserver started at http://127.23.222.65:46577/ using document root <none> and password file <none>
I20260929 15:58:44.519992 1381 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:44.520119 1381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:44.520481 1381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:44.523516 1381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "9e71dd4c7b114aa1b3a6099b4b841462"
format_stamp: "Formatted at 2026-09-29 15:58:44 on dist-test-slave-cq88"
server_key: "98f5f8baa72500a131f7fe18276c1ea9"
server_key_iv: "ec5fcab73e246a637b8ea1867df5a40c"
server_key_version: "encryptionkey@0"
I20260929 15:58:44.524371 1381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "9e71dd4c7b114aa1b3a6099b4b841462"
format_stamp: "Formatted at 2026-09-29 15:58:44 on dist-test-slave-cq88"
server_key: "98f5f8baa72500a131f7fe18276c1ea9"
server_key_iv: "ec5fcab73e246a637b8ea1867df5a40c"
server_key_version: "encryptionkey@0"
I20260929 15:58:44.529928 1381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:44.533808 1395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:44.535305 1381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 15:58:44.535493 1381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "9e71dd4c7b114aa1b3a6099b4b841462"
format_stamp: "Formatted at 2026-09-29 15:58:44 on dist-test-slave-cq88"
server_key: "98f5f8baa72500a131f7fe18276c1ea9"
server_key_iv: "ec5fcab73e246a637b8ea1867df5a40c"
server_key_version: "encryptionkey@0"
I20260929 15:58:44.535739 1381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:44.554311 1381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:44.555181 1381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:44.555540 1381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:44.556528 1381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:44.557955 1381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:44.558082 1381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:44.558195 1381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:44.558290 1381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:44.592631 1381 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:34339
I20260929 15:58:44.592645 1509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:34339 every 8 connection(s)
I20260929 15:58:44.594035 1381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:44.603181 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 1381
I20260929 15:58:44.603412 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:44.604223 24441 external_mini_cluster.cc:1428] Setting key b2dfd2908d0f2a8b1bddd4320d463483
I20260929 15:58:44.607038 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38387
--builtin_ntp_servers=127.23.222.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 15:58:44.607579 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:44.596620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:53567 (local address 127.23.222.126:38387)
0929 15:58:44.597147 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:44.597152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:44.598350 (+ 1198us) server_negotiation.cc:407] Connection header received
0929 15:58:44.600191 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:44.600197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:44.600287 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:44.600409 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:44.601483 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.602223 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.604579 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.605018 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.605158 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:44.606709 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:44.606713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:44.606717 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:44.606734 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:44.606769 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:44.606780 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:44.606875 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:44.607100 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 15:58:44.607194 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":262,"threads_started":1}
I20260929 15:58:44.608604 1513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:44.597277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38387 (local address 127.23.222.65:53567)
0929 15:58:44.598176 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:44.598218 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:44.599904 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:44.600640 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:44.600648 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:44.600736 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:44.601307 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:44.601322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.602376 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.602388 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:44.604432 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:44.604449 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:44.605251 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:44.605258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:44.605550 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:44.606373 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:44.606376 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:44.606417 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:44.606513 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:44.606875 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:44.606888 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:44.606899 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:58:44.607093 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 15:58:44.607510 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":265,"threads_started":1}
I20260929 15:58:44.610127 1510 heartbeater.cc:347] Connected to a master server at 127.23.222.126:38387
I20260929 15:58:44.610433 1510 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:44.611209 1510 heartbeater.cc:511] Master 127.23.222.126:38387 requested a full tablet report, sending...
I20260929 15:58:44.613469 1322 ts_manager.cc:194] Registered new tserver with Master: 9e71dd4c7b114aa1b3a6099b4b841462 (127.23.222.65:34339)
I20260929 15:58:44.614984 1322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.65:53567
W20260929 15:58:44.937160 1514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:44.937457 1514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:44.937644 1514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:44.945223 1514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:44.945338 1514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:44.945478 1514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:58:44.954816 1514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:44.956310 1514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:44.958174 1514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:44.971850 1519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:44.971932 1522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:44.971907 1520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:44.972591 1514 server_base.cc:1034] running on GCE node
I20260929 15:58:44.973405 1514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:44.974466 1514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:44.975697 1514 hybrid_clock.cc:648] HybridClock initialized: now 1790697524975646 us; error 50 us; skew 500 ppm
I20260929 15:58:44.976078 1514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:44.978519 1514 webserver.cc:460] Webserver started at http://127.23.222.66:44405/ using document root <none> and password file <none>
I20260929 15:58:44.979374 1514 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:44.979523 1514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:44.979907 1514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:44.982971 1514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "6be843c754ca41af9c628857f42a335a"
format_stamp: "Formatted at 2026-09-29 15:58:44 on dist-test-slave-cq88"
server_key: "4e3b59685ca4e53b1012e0db824dd527"
server_key_iv: "f4b6874b8038be5ff79169255c5898d1"
server_key_version: "encryptionkey@0"
I20260929 15:58:44.983973 1514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "6be843c754ca41af9c628857f42a335a"
format_stamp: "Formatted at 2026-09-29 15:58:44 on dist-test-slave-cq88"
server_key: "4e3b59685ca4e53b1012e0db824dd527"
server_key_iv: "f4b6874b8038be5ff79169255c5898d1"
server_key_version: "encryptionkey@0"
I20260929 15:58:44.989511 1514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:44.993211 1528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:44.994676 1514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:44.994902 1514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "6be843c754ca41af9c628857f42a335a"
format_stamp: "Formatted at 2026-09-29 15:58:44 on dist-test-slave-cq88"
server_key: "4e3b59685ca4e53b1012e0db824dd527"
server_key_iv: "f4b6874b8038be5ff79169255c5898d1"
server_key_version: "encryptionkey@0"
I20260929 15:58:44.995146 1514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:45.013411 1514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:45.014300 1514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:45.014617 1514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:45.015687 1514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:45.017694 1514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:45.017807 1514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:45.017962 1514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:45.018045 1514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:45.057243 1514 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:42843
I20260929 15:58:45.057300 1641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:42843 every 8 connection(s)
I20260929 15:58:45.058586 1514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:45.061903 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 1514
I20260929 15:58:45.062103 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:45.062682 24441 external_mini_cluster.cc:1428] Setting key 64117342768ecf113a38caf1a867ff0d
I20260929 15:58:45.065802 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38387
--builtin_ntp_servers=127.23.222.84:45967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 15:58:45.073266 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.061231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:33131 (local address 127.23.222.126:38387)
0929 15:58:45.061407 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:45.061411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:45.062993 (+ 1582us) server_negotiation.cc:407] Connection header received
0929 15:58:45.065184 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:45.065188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:45.065264 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:45.065351 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:45.066799 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.067622 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.070204 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.070601 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.070699 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.072421 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:45.072426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:45.072430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:45.072449 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:45.072511 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:45.072531 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:45.072617 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:45.072875 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 15:58:45.072986 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 15:58:45.074467 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.061676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38387 (local address 127.23.222.66:33131)
0929 15:58:45.062804 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:45.062854 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:45.064880 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:45.065625 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:45.065639 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:45.065758 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:45.066588 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.066615 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.067777 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.067785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.070033 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.070059 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.070835 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.070844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.071184 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.072031 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:45.072037 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:45.072091 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:45.072196 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:45.072640 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:45.072653 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:45.072664 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:58:45.072857 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 15:58:45.073354 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":323,"threads_started":1}
I20260929 15:58:45.076001 1642 heartbeater.cc:347] Connected to a master server at 127.23.222.126:38387
I20260929 15:58:45.076320 1642 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:45.077107 1642 heartbeater.cc:511] Master 127.23.222.126:38387 requested a full tablet report, sending...
I20260929 15:58:45.078594 1322 ts_manager.cc:194] Registered new tserver with Master: 6be843c754ca41af9c628857f42a335a (127.23.222.66:42843)
I20260929 15:58:45.079891 1322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.66:33131
W20260929 15:58:45.399102 1645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:45.399457 1645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:45.399560 1645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:45.406796 1645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:45.406898 1645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:45.407001 1645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:58:45.416240 1645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:45.417474 1645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:45.419260 1645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:45.432718 1651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:45.432667 1650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:45.433394 1653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:45.433657 1645 server_base.cc:1034] running on GCE node
I20260929 15:58:45.434288 1645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:45.435424 1645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:45.436663 1645 hybrid_clock.cc:648] HybridClock initialized: now 1790697525436605 us; error 52 us; skew 500 ppm
I20260929 15:58:45.437005 1645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:45.439492 1645 webserver.cc:460] Webserver started at http://127.23.222.67:35411/ using document root <none> and password file <none>
I20260929 15:58:45.440214 1645 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:45.440359 1645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:45.440771 1645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:45.443760 1645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "65b757ce0c1f4a43aed68c86df141de8"
format_stamp: "Formatted at 2026-09-29 15:58:45 on dist-test-slave-cq88"
server_key: "9377100e06a30b63efb22cfb4b55e011"
server_key_iv: "927e69f59785c76d67dcb8aa758440af"
server_key_version: "encryptionkey@0"
I20260929 15:58:45.444571 1645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "65b757ce0c1f4a43aed68c86df141de8"
format_stamp: "Formatted at 2026-09-29 15:58:45 on dist-test-slave-cq88"
server_key: "9377100e06a30b63efb22cfb4b55e011"
server_key_iv: "927e69f59785c76d67dcb8aa758440af"
server_key_version: "encryptionkey@0"
I20260929 15:58:45.450030 1645 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:58:45.453991 1659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:45.455658 1645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:45.455856 1645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "65b757ce0c1f4a43aed68c86df141de8"
format_stamp: "Formatted at 2026-09-29 15:58:45 on dist-test-slave-cq88"
server_key: "9377100e06a30b63efb22cfb4b55e011"
server_key_iv: "927e69f59785c76d67dcb8aa758440af"
server_key_version: "encryptionkey@0"
I20260929 15:58:45.456120 1645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:45.474443 1645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:45.475397 1645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:45.475728 1645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:45.476722 1645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:45.478209 1645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:45.478336 1645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:45.478477 1645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:45.478577 1645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:45.513006 1645 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:38227
I20260929 15:58:45.513078 1774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:38227 every 8 connection(s)
I20260929 15:58:45.514339 1645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:45.522236 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 1645
I20260929 15:58:45.522418 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:45.522953 24441 external_mini_cluster.cc:1428] Setting key b95d3a242c892149c59806d1617fca3b
I20260929 15:58:45.527868 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.517050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:57127 (local address 127.23.222.126:38387)
0929 15:58:45.517246 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:45.517252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:45.518641 (+ 1389us) server_negotiation.cc:407] Connection header received
0929 15:58:45.520547 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:45.520553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:45.520644 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:45.520756 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:45.521787 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.522934 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.524857 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.525504 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.525631 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.527036 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:45.527042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:45.527045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:45.527064 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:45.527098 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:45.527111 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:45.527198 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:45.527498 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 15:58:45.527579 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:58:45.528838 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.517554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38387 (local address 127.23.222.67:57127)
0929 15:58:45.518475 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:45.518517 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:45.520271 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:45.520983 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:45.520991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:45.521074 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:45.521625 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.521640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.523143 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.523147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.524713 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:45.524725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.525721 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.525725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:45.526020 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.526676 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:45.526679 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:45.526717 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:45.526792 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:45.527209 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:45.527216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:45.527222 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:58:45.527459 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 15:58:45.527759 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":290,"threads_started":1}
I20260929 15:58:45.530297 1775 heartbeater.cc:347] Connected to a master server at 127.23.222.126:38387
I20260929 15:58:45.530736 1775 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:45.531687 1775 heartbeater.cc:511] Master 127.23.222.126:38387 requested a full tablet report, sending...
I20260929 15:58:45.533139 1322 ts_manager.cc:194] Registered new tserver with Master: 65b757ce0c1f4a43aed68c86df141de8 (127.23.222.67:38227)
I20260929 15:58:45.533926 1322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.67:57127
I20260929 15:58:45.538357 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:58:45.544546 1512 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:45.542994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.23.222.126:38387)
0929 15:58:45.543157 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:45.543161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:45.543668 (+ 507us) server_negotiation.cc:407] Connection header received
0929 15:58:45.543809 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:45.543813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:45.543894 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:45.544002 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:45.544369 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45626: BlockingRecv error: recv got EOF from 127.0.0.1:45626 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 15:58:45.545491 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 1381
I20260929 15:58:45.553340 1784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.546335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.23.222.65:34339)
0929 15:58:45.546942 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:45.546949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:45.546975 (+ 26us) server_negotiation.cc:407] Connection header received
0929 15:58:45.547025 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:45.547031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:45.547320 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:45.547487 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:45.548093 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.549090 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.550348 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.550875 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.550973 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.552036 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:45.552043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:45.552055 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:45.552637 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:45.552683 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:45.552702 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:45.552791 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:45.552917 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 15:58:45.553057 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":301,"threads_started":1}
I20260929 15:58:45.554595 1505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:45.636572 1510 heartbeater.cc:503] Master 127.23.222.126:38387 was elected leader, sending a full tablet report...
I20260929 15:58:45.637974 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 1381
W20260929 15:58:45.655406 1308 connection.cc:535] server connection from 127.23.222.65:53567 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 15:58:45.655543 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 1514
I20260929 15:58:45.663592 1786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.656549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.23.222.66:42843)
0929 15:58:45.657112 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:45.657119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:45.657147 (+ 28us) server_negotiation.cc:407] Connection header received
0929 15:58:45.657213 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:45.657220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:45.657439 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:45.657581 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:45.657914 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.658893 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.660228 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.660850 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.660967 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.661949 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:45.661958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:45.661972 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:45.662592 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:45.662638 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:45.662664 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:45.662770 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:45.662936 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 15:58:45.663098 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":280,"threads_started":1}
I20260929 15:58:45.664870 1637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:45.747682 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 1514
I20260929 15:58:45.764065 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 1645
I20260929 15:58:45.772382 1788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:45.764988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.23.222.67:38227)
0929 15:58:45.765597 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:45.765604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:45.765636 (+ 32us) server_negotiation.cc:407] Connection header received
0929 15:58:45.765696 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:45.765702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:45.765934 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:45.766064 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:45.766474 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.767612 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.768916 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:45.769649 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:45.769815 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:45.770815 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:45.770824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:45.770837 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:45.771534 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:45.771582 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:45.771602 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:45.771706 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:45.771840 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 15:58:45.771978 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":295,"threads_started":1}
I20260929 15:58:45.773778 1770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:45.854667 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 1645
I20260929 15:58:45.870498 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 1290
I20260929 15:58:45.871825 1352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:45.975840 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 1290
2026-09-29T15:58:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2473 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T15:58:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:46Z Disabled control of system clock
I20260929 15:58:46.050161 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:40675
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:40675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 15:58:46.382982 1794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:46.383390 1794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:46.383540 1794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:46.391522 1794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:46.391644 1794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:46.391727 1794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:46.391829 1794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:46.391911 1794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:46.401419 1794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:40675
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:40675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:46.402772 1794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:46.404680 1794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:46.416263 1799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:46.416452 1802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:46.416725 1800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:46.417574 1794 server_base.cc:1034] running on GCE node
I20260929 15:58:46.418339 1794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:46.419837 1794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:46.421090 1794 hybrid_clock.cc:648] HybridClock initialized: now 1790697526421025 us; error 81 us; skew 500 ppm
I20260929 15:58:46.421485 1794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:46.423899 1794 webserver.cc:460] Webserver started at http://127.23.222.126:40585/ using document root <none> and password file <none>
I20260929 15:58:46.424619 1794 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:46.424751 1794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:46.425083 1794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:46.428016 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "0777bff0397f43959d8bf9ba275cb4a0"
format_stamp: "Formatted at 2026-09-29 15:58:46 on dist-test-slave-cq88"
server_key: "33ecfaa224346d79cd541fa49fd6136e"
server_key_iv: "c08da2712564f73434eafabb6c40309a"
server_key_version: "encryptionkey@0"
I20260929 15:58:46.428829 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "0777bff0397f43959d8bf9ba275cb4a0"
format_stamp: "Formatted at 2026-09-29 15:58:46 on dist-test-slave-cq88"
server_key: "33ecfaa224346d79cd541fa49fd6136e"
server_key_iv: "c08da2712564f73434eafabb6c40309a"
server_key_version: "encryptionkey@0"
I20260929 15:58:46.434047 1794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 15:58:46.437649 1809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:46.439049 1794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 15:58:46.439296 1794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "0777bff0397f43959d8bf9ba275cb4a0"
format_stamp: "Formatted at 2026-09-29 15:58:46 on dist-test-slave-cq88"
server_key: "33ecfaa224346d79cd541fa49fd6136e"
server_key_iv: "c08da2712564f73434eafabb6c40309a"
server_key_version: "encryptionkey@0"
I20260929 15:58:46.439528 1794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:46.445339 1794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:46.446173 1794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:46.446493 1794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:46.463904 1794 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:40675
I20260929 15:58:46.463922 1860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:40675 every 8 connection(s)
I20260929 15:58:46.465341 1794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:46.469285 1861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:46.472301 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 1794
I20260929 15:58:46.472518 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:58:46.473160 24441 external_mini_cluster.cc:1428] Setting key 19c6d0880e1e4753e77e358eb5fc3944
I20260929 15:58:46.482856 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:46.475404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.23.222.126:40675)
0929 15:58:46.476589 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:46.476613 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:46.476678 (+ 65us) server_negotiation.cc:407] Connection header received
0929 15:58:46.478617 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:46.478653 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:46.479309 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:46.479806 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:46.480862 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:46.480871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:46.480894 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:46.480912 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:46.480982 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:46.481005 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:46.481238 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:46.481370 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 15:58:46.481625 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":368,"threads_started":1}
I20260929 15:58:46.483387 1861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0: Bootstrap starting.
I20260929 15:58:46.486702 1861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:46.487903 1861 log.cc:824] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:46.490585 1861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0: No bootstrap required, opened a new log
I20260929 15:58:46.497656 1861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0777bff0397f43959d8bf9ba275cb4a0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40675 } }
I20260929 15:58:46.498029 1861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:46.498124 1861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0777bff0397f43959d8bf9ba275cb4a0, State: Initialized, Role: FOLLOWER
I20260929 15:58:46.498797 1861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0777bff0397f43959d8bf9ba275cb4a0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40675 } }
I20260929 15:58:46.499050 1861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:46.499199 1861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:46.499398 1861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:46.501489 1861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0777bff0397f43959d8bf9ba275cb4a0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40675 } }
I20260929 15:58:46.501992 1861 leader_election.cc:302] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0777bff0397f43959d8bf9ba275cb4a0; no voters:
I20260929 15:58:46.502625 1861 leader_election.cc:288] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:46.502768 1866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:46.503530 1866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [term 1 LEADER]: Becoming Leader. State: Replica: 0777bff0397f43959d8bf9ba275cb4a0, State: Running, Role: LEADER
I20260929 15:58:46.504151 1866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0777bff0397f43959d8bf9ba275cb4a0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40675 } }
I20260929 15:58:46.504745 1861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:46.506908 1867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0777bff0397f43959d8bf9ba275cb4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0777bff0397f43959d8bf9ba275cb4a0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40675 } } }
I20260929 15:58:46.507095 1868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0777bff0397f43959d8bf9ba275cb4a0. Latest consensus state: current_term: 1 leader_uuid: "0777bff0397f43959d8bf9ba275cb4a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0777bff0397f43959d8bf9ba275cb4a0" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40675 } } }
I20260929 15:58:46.507344 1867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:46.507412 1868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:46.508849 1874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:46.514133 1874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:46.522924 1874 catalog_manager.cc:1347] Generated new cluster ID: 9e41d61a60404121a709ad186d3645d9
I20260929 15:58:46.523066 1874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:46.531332 1874 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:46.531760 1874 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:46.543216 1874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0777bff0397f43959d8bf9ba275cb4a0: Generated new TSK 0
I20260929 15:58:46.544273 1874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 15:58:46.552574 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:40675
--builtin_ntp_servers=127.23.222.84:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 15:58:46.898554 1886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:46.898877 1886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:46.899055 1886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:46.907080 1886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:46.907207 1886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:46.907423 1886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:58:46.917244 1886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:40675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:46.918671 1886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:46.920588 1886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:46.934441 1894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:46.934383 1891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:46.934415 1892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:46.935570 1886 server_base.cc:1034] running on GCE node
I20260929 15:58:46.936177 1886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:46.937192 1886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:46.938431 1886 hybrid_clock.cc:648] HybridClock initialized: now 1790697526938384 us; error 44 us; skew 500 ppm
I20260929 15:58:46.938786 1886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:46.941283 1886 webserver.cc:460] Webserver started at http://127.23.222.65:44325/ using document root <none> and password file <none>
I20260929 15:58:46.942018 1886 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:46.942139 1886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:46.942492 1886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:46.945613 1886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "237461b9a1df441c9fda8045b492449d"
format_stamp: "Formatted at 2026-09-29 15:58:46 on dist-test-slave-cq88"
server_key: "41f8f8a221b9361ffd8589dd7c457073"
server_key_iv: "98ff2ccaf13a7f6d1371a0d16f56b76a"
server_key_version: "encryptionkey@0"
I20260929 15:58:46.946434 1886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "237461b9a1df441c9fda8045b492449d"
format_stamp: "Formatted at 2026-09-29 15:58:46 on dist-test-slave-cq88"
server_key: "41f8f8a221b9361ffd8589dd7c457073"
server_key_iv: "98ff2ccaf13a7f6d1371a0d16f56b76a"
server_key_version: "encryptionkey@0"
I20260929 15:58:46.951992 1886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:58:46.955619 1900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:46.957074 1886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:58:46.957229 1886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "237461b9a1df441c9fda8045b492449d"
format_stamp: "Formatted at 2026-09-29 15:58:46 on dist-test-slave-cq88"
server_key: "41f8f8a221b9361ffd8589dd7c457073"
server_key_iv: "98ff2ccaf13a7f6d1371a0d16f56b76a"
server_key_version: "encryptionkey@0"
I20260929 15:58:46.957427 1886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:46.963150 1886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:46.963963 1886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:46.964188 1886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:46.965036 1886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:46.966362 1886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:46.966449 1886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:46.966569 1886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:46.966627 1886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:47.000420 1886 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:36101
I20260929 15:58:47.000495 2012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:36101 every 8 connection(s)
I20260929 15:58:47.001741 1886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:47.006356 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 1886
I20260929 15:58:47.006573 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:47.007190 24441 external_mini_cluster.cc:1428] Setting key 6bd2d2880b931c35d7afa3f7566f5a59
I20260929 15:58:47.010324 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:40675
--builtin_ntp_servers=127.23.222.84:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 15:58:47.010635 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.004482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:35711 (local address 127.23.222.126:40675)
0929 15:58:47.004970 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:47.004975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:47.006225 (+ 1250us) server_negotiation.cc:407] Connection header received
0929 15:58:47.008187 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:47.008192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:47.008270 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:47.008413 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:47.009822 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:47.009828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:47.009833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:47.009854 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:47.009910 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:47.009930 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:47.010058 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:47.010232 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 15:58:47.010300 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":256,"threads_started":1}
I20260929 15:58:47.011724 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.004988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:40675 (local address 127.23.222.65:35711)
0929 15:58:47.006055 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:47.006102 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:47.007886 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:47.008632 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:47.008640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:47.008735 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:47.009515 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:47.009517 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:47.009556 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:47.009642 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:47.010031 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:47.010038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:47.010044 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 15:58:47.010255 (+ 211us) client_negotiation.cc:240] Negotiation successful
0929 15:58:47.010529 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":266,"threads_started":1}
I20260929 15:58:47.013293 2013 heartbeater.cc:347] Connected to a master server at 127.23.222.126:40675
I20260929 15:58:47.013616 2013 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:47.014323 2013 heartbeater.cc:511] Master 127.23.222.126:40675 requested a full tablet report, sending...
I20260929 15:58:47.016636 1826 ts_manager.cc:194] Registered new tserver with Master: 237461b9a1df441c9fda8045b492449d (127.23.222.65:36101)
W20260929 15:58:47.349479 2017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:47.349763 2017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:47.349853 2017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:47.357770 2017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:47.357888 2017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:47.358088 2017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:58:47.367884 2017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:40675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:47.369199 2017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:47.371037 2017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:47.383737 2023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:47.383733 2022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:47.384665 2026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:47.385977 2017 server_base.cc:1034] running on GCE node
I20260929 15:58:47.386674 2017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:47.387724 2017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:47.389011 2017 hybrid_clock.cc:648] HybridClock initialized: now 1790697527388960 us; error 54 us; skew 500 ppm
I20260929 15:58:47.389415 2017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:47.391892 2017 webserver.cc:460] Webserver started at http://127.23.222.66:36209/ using document root <none> and password file <none>
I20260929 15:58:47.392594 2017 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:47.392726 2017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:47.393079 2017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:47.397073 2017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "702659773724451a939503e60522802d"
format_stamp: "Formatted at 2026-09-29 15:58:47 on dist-test-slave-cq88"
server_key: "d53afdd7aafd74dd1f99202bcd76b209"
server_key_iv: "f22e5fd463575b023eb07a8b4a131066"
server_key_version: "encryptionkey@0"
I20260929 15:58:47.398185 2017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "702659773724451a939503e60522802d"
format_stamp: "Formatted at 2026-09-29 15:58:47 on dist-test-slave-cq88"
server_key: "d53afdd7aafd74dd1f99202bcd76b209"
server_key_iv: "f22e5fd463575b023eb07a8b4a131066"
server_key_version: "encryptionkey@0"
I20260929 15:58:47.404583 2017 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 15:58:47.408182 2032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:47.409529 2017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 15:58:47.409693 2017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "702659773724451a939503e60522802d"
format_stamp: "Formatted at 2026-09-29 15:58:47 on dist-test-slave-cq88"
server_key: "d53afdd7aafd74dd1f99202bcd76b209"
server_key_iv: "f22e5fd463575b023eb07a8b4a131066"
server_key_version: "encryptionkey@0"
I20260929 15:58:47.409891 2017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:47.415930 2017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:47.416781 2017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:47.417039 2017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:47.417981 2017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:47.419380 2017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:47.419487 2017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:47.419613 2017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:47.419687 2017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:47.453963 2017 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:38723
I20260929 15:58:47.454066 2144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:38723 every 8 connection(s)
I20260929 15:58:47.455372 2017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:47.456718 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 2017
I20260929 15:58:47.456912 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:47.457353 24441 external_mini_cluster.cc:1428] Setting key ff10d7fd80d75ef735b30a01e75c9823
I20260929 15:58:47.460240 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:40675
--builtin_ntp_servers=127.23.222.84:39933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 15:58:47.464757 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.458334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:52475 (local address 127.23.222.126:40675)
0929 15:58:47.458540 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:47.458545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:47.460232 (+ 1687us) server_negotiation.cc:407] Connection header received
0929 15:58:47.462390 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:47.462395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:47.462466 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:47.462591 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:47.463965 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:47.463971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:47.463975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:47.463996 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:47.464042 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:47.464055 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:47.464188 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:47.464388 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 15:58:47.464459 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 15:58:47.465818 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.458936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:40675 (local address 127.23.222.66:52475)
0929 15:58:47.460069 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:47.460120 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:47.462076 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:47.462831 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:47.462840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:47.462938 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:47.463665 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:47.463667 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:47.463706 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:47.463793 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:47.464154 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:47.464164 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:47.464171 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 15:58:47.464472 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 15:58:47.464739 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":390,"threads_started":1}
I20260929 15:58:47.467196 2145 heartbeater.cc:347] Connected to a master server at 127.23.222.126:40675
I20260929 15:58:47.467592 2145 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:47.468323 2145 heartbeater.cc:511] Master 127.23.222.126:40675 requested a full tablet report, sending...
I20260929 15:58:47.469905 1826 ts_manager.cc:194] Registered new tserver with Master: 702659773724451a939503e60522802d (127.23.222.66:38723)
W20260929 15:58:47.788789 2148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:47.789074 2148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:47.789168 2148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:47.796468 2148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:47.796566 2148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:47.796679 2148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:58:47.805733 2148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:40675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:47.807045 2148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:47.808830 2148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:47.821869 2153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:47.821892 2156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:47.822338 2154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:47.822660 2148 server_base.cc:1034] running on GCE node
I20260929 15:58:47.823365 2148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:47.824335 2148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:47.825538 2148 hybrid_clock.cc:648] HybridClock initialized: now 1790697527825485 us; error 43 us; skew 500 ppm
I20260929 15:58:47.825907 2148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 15:58:47.828280 2148 webserver.cc:460] Webserver started at http://127.23.222.67:43091/ using document root <none> and password file <none>
I20260929 15:58:47.828926 2148 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:47.829020 2148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:47.829281 2148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:47.832181 2148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "e1af7336d8c84b05a060d271ddaec6af"
format_stamp: "Formatted at 2026-09-29 15:58:47 on dist-test-slave-cq88"
server_key: "16bd68142e2fc085891e4ed26ecd7454"
server_key_iv: "d6fce167743566f12a57e5566d85f1a7"
server_key_version: "encryptionkey@0"
I20260929 15:58:47.832913 2148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "e1af7336d8c84b05a060d271ddaec6af"
format_stamp: "Formatted at 2026-09-29 15:58:47 on dist-test-slave-cq88"
server_key: "16bd68142e2fc085891e4ed26ecd7454"
server_key_iv: "d6fce167743566f12a57e5566d85f1a7"
server_key_version: "encryptionkey@0"
I20260929 15:58:47.838119 2148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 15:58:47.841501 2162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:47.842900 2148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 15:58:47.843047 2148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "e1af7336d8c84b05a060d271ddaec6af"
format_stamp: "Formatted at 2026-09-29 15:58:47 on dist-test-slave-cq88"
server_key: "16bd68142e2fc085891e4ed26ecd7454"
server_key_iv: "d6fce167743566f12a57e5566d85f1a7"
server_key_version: "encryptionkey@0"
I20260929 15:58:47.843230 2148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:47.848848 2148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:47.849601 2148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:47.849825 2148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:47.850693 2148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:47.852128 2148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:47.852216 2148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:47.852294 2148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:47.852349 2148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:47.885996 2148 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:44933
I20260929 15:58:47.886044 2275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:44933 every 8 connection(s)
I20260929 15:58:47.887425 2148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:47.894881 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 2148
I20260929 15:58:47.895125 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:47.895617 24441 external_mini_cluster.cc:1428] Setting key 3c97423e0405eaafa33464f844e75e7e
I20260929 15:58:47.896421 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.890190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:49573 (local address 127.23.222.126:40675)
0929 15:58:47.890399 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:47.890404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:47.892057 (+ 1653us) server_negotiation.cc:407] Connection header received
0929 15:58:47.893803 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:47.893807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:47.893868 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:47.893971 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:47.895595 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:47.895602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:47.895611 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:47.895630 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:47.895664 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:47.895675 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:47.895834 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:47.896074 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 15:58:47.896127 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:58:47.897797 2278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.890760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:40675 (local address 127.23.222.67:49573)
0929 15:58:47.891864 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:47.891905 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:47.893491 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:47.894197 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:47.894206 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:47.894306 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:47.895207 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 15:58:47.895213 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:47.895302 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 15:58:47.895401 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:47.895814 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:47.895826 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:47.895837 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 15:58:47.896166 (+ 329us) client_negotiation.cc:240] Negotiation successful
0929 15:58:47.896418 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":312,"threads_started":1}
I20260929 15:58:47.899480 2276 heartbeater.cc:347] Connected to a master server at 127.23.222.126:40675
I20260929 15:58:47.899811 2276 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:47.900424 2276 heartbeater.cc:511] Master 127.23.222.126:40675 requested a full tablet report, sending...
I20260929 15:58:47.901888 1825 ts_manager.cc:194] Registered new tserver with Master: e1af7336d8c84b05a060d271ddaec6af (127.23.222.67:44933)
I20260929 15:58:47.911932 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 15:58:47.918368 2015 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 15:58:47.916730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.23.222.126:40675)
0929 15:58:47.917004 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:47.917009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:47.917511 (+ 502us) server_negotiation.cc:407] Connection header received
0929 15:58:47.917639 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:47.917643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:47.917699 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:47.917802 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:47.918071 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54930: BlockingRecv error: recv got EOF from 127.0.0.1:54930 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 15:58:47.919144 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 1886
I20260929 15:58:47.922557 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:47.920145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.23.222.65:36101)
0929 15:58:47.920642 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:47.920649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:47.920674 (+ 25us) server_negotiation.cc:407] Connection header received
0929 15:58:47.920730 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:47.920737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:47.920951 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:47.921075 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:47.921932 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:47.921939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:47.921951 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:47.921970 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:47.922014 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:47.922028 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:47.922130 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:47.922212 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 15:58:47.922318 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":255,"threads_started":1}
I20260929 15:58:47.923784 2008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:48.006455 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 1886
I20260929 15:58:48.023197 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 2017
I20260929 15:58:48.026746 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:48.024173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56342 (local address 127.23.222.66:38723)
0929 15:58:48.024684 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:48.024692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:48.024722 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:58:48.024788 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:48.024795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:48.025022 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:48.025153 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:48.026011 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:48.026017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:48.026029 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:48.026041 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:48.026080 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:48.026131 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:48.026280 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:48.026429 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 15:58:48.026518 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260929 15:58:48.028262 2140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:48.110499 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 2017
I20260929 15:58:48.126775 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 2148
I20260929 15:58:48.130511 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:48.127863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46416 (local address 127.23.222.67:44933)
0929 15:58:48.128530 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:48.128537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:48.128567 (+ 30us) server_negotiation.cc:407] Connection header received
0929 15:58:48.128637 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:48.128643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:48.128858 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:48.128996 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:48.129889 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:48.129895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:48.129907 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 15:58:48.129923 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:48.129960 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 15:58:48.129973 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:48.130119 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:48.130170 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 15:58:48.130249 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":345,"threads_started":1}
I20260929 15:58:48.131891 2271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:48.219297 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 2148
I20260929 15:58:48.235584 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 1794
I20260929 15:58:48.236840 1856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:48.338492 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 1794
2026-09-29T15:58:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2362 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:48 dist-test-slave-cq88 krb5kdc[2294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:48 dist-test-slave-cq88 krb5kdc[2294](info): set up 2 sockets
Sep 29 15:58:48 dist-test-slave-cq88 krb5kdc[2294](info): commencing operation
krb5kdc: starting...
W20260929 15:58:49.495471 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:58:49 dist-test-slave-cq88 krb5kdc[2294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:58:49.739652 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:39669
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:39669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:50.086654 2307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:50.087018 2307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:50.087184 2307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:50.094857 2307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:58:50.095019 2307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:50.095140 2307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:50.095307 2307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:58:50.095399 2307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:58:50.105713 2307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:39669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:39669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:50.107120 2307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:50.108469 2307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:50.120328 2313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:50.120297 2316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:50.120379 2314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:50.121538 2307 server_base.cc:1034] running on GCE node
I20260929 15:58:50.122453 2307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:50.123992 2307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:50.125236 2307 hybrid_clock.cc:648] HybridClock initialized: now 1790697530125189 us; error 39 us; skew 500 ppm
I20260929 15:58:50.125628 2307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:50 dist-test-slave-cq88 krb5kdc[2294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:50.128832 2307 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:58:50.130404 2307 webserver.cc:460] Webserver started at http://127.23.222.126:37837/ using document root <none> and password file <none>
I20260929 15:58:50.131172 2307 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:50.131354 2307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:50.131669 2307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:50.134583 2307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "4910c65495da44b6be90408c8f6efbc9"
format_stamp: "Formatted at 2026-09-29 15:58:50 on dist-test-slave-cq88"
server_key: "a3adf2d162dad3f43431edf5761aac3e"
server_key_iv: "ac64b6d14a1fda0fae2f64ff4aa7fad1"
server_key_version: "encryptionkey@0"
I20260929 15:58:50.135435 2307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "4910c65495da44b6be90408c8f6efbc9"
format_stamp: "Formatted at 2026-09-29 15:58:50 on dist-test-slave-cq88"
server_key: "a3adf2d162dad3f43431edf5761aac3e"
server_key_iv: "ac64b6d14a1fda0fae2f64ff4aa7fad1"
server_key_version: "encryptionkey@0"
I20260929 15:58:50.141065 2307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:50.144520 2323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:50.145835 2307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 15:58:50.146020 2307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "4910c65495da44b6be90408c8f6efbc9"
format_stamp: "Formatted at 2026-09-29 15:58:50 on dist-test-slave-cq88"
server_key: "a3adf2d162dad3f43431edf5761aac3e"
server_key_iv: "ac64b6d14a1fda0fae2f64ff4aa7fad1"
server_key_version: "encryptionkey@0"
I20260929 15:58:50.146229 2307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:50.165690 2307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:50.169835 2307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:50.170151 2307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:50.194634 2307 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:39669
I20260929 15:58:50.194655 2374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:39669 every 8 connection(s)
I20260929 15:58:50.196301 2307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:58:50.200317 2375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:50.203696 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 2307
I20260929 15:58:50.203962 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:58:50.204823 24441 external_mini_cluster.cc:1428] Setting key 8987d8fb48f0f9de1e1bc7df5c308614
I20260929 15:58:50.214051 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9: Bootstrap starting.
I20260929 15:58:50.218183 2375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:50.219676 2375 log.cc:824] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9: Log is configured to *not* fsync() on all Append() calls
Sep 29 15:58:50 dist-test-slave-cq88 krb5kdc[2294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:50.223080 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9: No bootstrap required, opened a new log
I20260929 15:58:50.228900 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:50.207011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44812 (local address 127.23.222.126:39669)
0929 15:58:50.207883 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:50.207895 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:50.207938 (+ 43us) server_negotiation.cc:407] Connection header received
0929 15:58:50.209562 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:50.209584 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:50.209990 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:50.210326 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:50.212320 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.213615 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.215529 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.216202 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.216341 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:50.221640 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:50.221649 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:50.221665 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:50.221705 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:50.224719 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:50.225453 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:50.225462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:50.225468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:50.225631 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:50.226059 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:50.226063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:50.226068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:50.226622 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:50.226950 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:50.227432 (+ 482us) server_negotiation.cc:299] Negotiation successful
0929 15:58:50.227865 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":477,"threads_started":1}
I20260929 15:58:50.231534 2375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4910c65495da44b6be90408c8f6efbc9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39669 } }
I20260929 15:58:50.232009 2375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:50.232120 2375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4910c65495da44b6be90408c8f6efbc9, State: Initialized, Role: FOLLOWER
I20260929 15:58:50.232654 2375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4910c65495da44b6be90408c8f6efbc9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39669 } }
I20260929 15:58:50.232859 2375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:58:50.232954 2375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:58:50.233088 2375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:50.235172 2375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4910c65495da44b6be90408c8f6efbc9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39669 } }
I20260929 15:58:50.235678 2375 leader_election.cc:302] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4910c65495da44b6be90408c8f6efbc9; no voters:
I20260929 15:58:50.236261 2375 leader_election.cc:288] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:58:50.236373 2380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:50.237115 2380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [term 1 LEADER]: Becoming Leader. State: Replica: 4910c65495da44b6be90408c8f6efbc9, State: Running, Role: LEADER
I20260929 15:58:50.237617 2380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4910c65495da44b6be90408c8f6efbc9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39669 } }
I20260929 15:58:50.238584 2375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:58:50.240937 2382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4910c65495da44b6be90408c8f6efbc9. Latest consensus state: current_term: 1 leader_uuid: "4910c65495da44b6be90408c8f6efbc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4910c65495da44b6be90408c8f6efbc9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39669 } } }
I20260929 15:58:50.241089 2381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4910c65495da44b6be90408c8f6efbc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4910c65495da44b6be90408c8f6efbc9" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39669 } } }
I20260929 15:58:50.241413 2381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:50.241418 2382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9 [sys.catalog]: This master's current role is: LEADER
I20260929 15:58:50.243165 2385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:58:50.248453 2385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:58:50.257568 2385 catalog_manager.cc:1347] Generated new cluster ID: acbb8d553fe54032a00decc812c2fd36
I20260929 15:58:50.257700 2385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:58:50.274216 2385 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:58:50.275728 2385 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:58:50.291152 2385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9: Generated new TSK 0
I20260929 15:58:50.292241 2385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:58:50.363336 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:39669
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:50.707603 2401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:50.707944 2401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:50.708109 2401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:50.715677 2401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:50.715819 2401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:50.715955 2401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:58:50.725972 2401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:39669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:50.727344 2401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:50.728855 2401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:50.742552 2409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:50.742698 2406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:50.742663 2407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:50.744012 2401 server_base.cc:1034] running on GCE node
I20260929 15:58:50.744737 2401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:50.745782 2401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:50.747038 2401 hybrid_clock.cc:648] HybridClock initialized: now 1790697530746976 us; error 39 us; skew 500 ppm
I20260929 15:58:50.747506 2401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:50 dist-test-slave-cq88 krb5kdc[2294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:50.751226 2401 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:58:50.752916 2401 webserver.cc:460] Webserver started at http://127.23.222.65:40457/ using document root <none> and password file <none>
I20260929 15:58:50.753649 2401 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:50.753806 2401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:50.754117 2401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:50.757118 2401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "d0af629e89a54d459357cb130eadf5b9"
format_stamp: "Formatted at 2026-09-29 15:58:50 on dist-test-slave-cq88"
server_key: "34b0950e5cea74f4f5b26c96a3923ade"
server_key_iv: "5e68c44833194d13fdebec26beace46c"
server_key_version: "encryptionkey@0"
I20260929 15:58:50.757886 2401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "d0af629e89a54d459357cb130eadf5b9"
format_stamp: "Formatted at 2026-09-29 15:58:50 on dist-test-slave-cq88"
server_key: "34b0950e5cea74f4f5b26c96a3923ade"
server_key_iv: "5e68c44833194d13fdebec26beace46c"
server_key_version: "encryptionkey@0"
I20260929 15:58:50.763195 2401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 15:58:50.766788 2416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:50.768261 2401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 15:58:50.768456 2401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "d0af629e89a54d459357cb130eadf5b9"
format_stamp: "Formatted at 2026-09-29 15:58:50 on dist-test-slave-cq88"
server_key: "34b0950e5cea74f4f5b26c96a3923ade"
server_key_iv: "5e68c44833194d13fdebec26beace46c"
server_key_version: "encryptionkey@0"
I20260929 15:58:50.768630 2401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:50.789104 2401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:50.793674 2401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:50.793938 2401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:50.794845 2401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:50.796272 2401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:50.796368 2401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:50.796458 2401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:50.796495 2401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:50.829945 2401 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:39099
I20260929 15:58:50.830035 2529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:39099 every 8 connection(s)
I20260929 15:58:50.831281 2401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:58:50.839998 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 2401
I20260929 15:58:50.840269 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:58:50.841109 24441 external_mini_cluster.cc:1428] Setting key 1e9abf2476c05ededf9846bc89b810f4
Sep 29 15:58:50 dist-test-slave-cq88 krb5kdc[2294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:50.853343 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:50.834098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:33269 (local address 127.23.222.126:39669)
0929 15:58:50.834617 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:50.834624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:50.835748 (+ 1124us) server_negotiation.cc:407] Connection header received
0929 15:58:50.837606 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:50.837611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:50.837689 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:50.837800 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:50.839425 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.840377 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.842557 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.843164 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.843340 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:50.847547 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:50.847552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:50.847555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:50.847586 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:50.850236 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:50.850982 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:50.850991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:50.850993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:50.851112 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:50.851573 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:50.851579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:50.851582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:50.851854 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:50.852010 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:50.852743 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 15:58:50.852969 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260929 15:58:50.854357 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:50.834756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:39669 (local address 127.23.222.65:33269)
0929 15:58:50.835587 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:50.835627 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:50.837332 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:50.838043 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:50.838051 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:50.838578 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:50.839134 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:50.839153 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.840553 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.840559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:50.842397 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:50.842409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:50.843420 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:50.843455 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:50.843749 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:50.844590 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:50.844625 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:50.847332 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:50.850384 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:50.850392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:50.850409 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:50.850848 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:50.851226 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:50.851231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:50.851257 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:50.851449 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:50.851991 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:50.851998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:50.852361 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 15:58:50.852620 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 15:58:50.853158 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":294,"threads_started":1}
I20260929 15:58:50.856055 2530 heartbeater.cc:347] Connected to a master server at 127.23.222.126:39669
I20260929 15:58:50.856398 2530 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:50.857147 2530 heartbeater.cc:511] Master 127.23.222.126:39669 requested a full tablet report, sending...
I20260929 15:58:50.859462 2340 ts_manager.cc:194] Registered new tserver with Master: d0af629e89a54d459357cb130eadf5b9 (127.23.222.65:39099)
I20260929 15:58:50.861699 2340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:33269
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:58:50.896953 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:39669
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:51.226310 2536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:51.226675 2536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:51.226864 2536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:51.234982 2536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:51.235169 2536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:51.235376 2536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:58:51.245895 2536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:39669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:51.247336 2536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:51.248672 2536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:51.262303 2541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:51.262230 2542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:51.262318 2544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:51.263342 2536 server_base.cc:1034] running on GCE node
I20260929 15:58:51.264001 2536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:51.265018 2536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:51.266261 2536 hybrid_clock.cc:648] HybridClock initialized: now 1790697531266219 us; error 60 us; skew 500 ppm
I20260929 15:58:51.266600 2536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:51 dist-test-slave-cq88 krb5kdc[2294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:51.269887 2536 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:58:51.271546 2536 webserver.cc:460] Webserver started at http://127.23.222.66:40401/ using document root <none> and password file <none>
I20260929 15:58:51.272321 2536 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:51.272466 2536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:51.272774 2536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:51.276000 2536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "d1fb9ac8dd804eda894764f99bf2a494"
format_stamp: "Formatted at 2026-09-29 15:58:51 on dist-test-slave-cq88"
server_key: "7405d35cae7d7d0dea648a23f08a1b5f"
server_key_iv: "28e72262dc18872f559e07da8b4cad58"
server_key_version: "encryptionkey@0"
I20260929 15:58:51.276757 2536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "d1fb9ac8dd804eda894764f99bf2a494"
format_stamp: "Formatted at 2026-09-29 15:58:51 on dist-test-slave-cq88"
server_key: "7405d35cae7d7d0dea648a23f08a1b5f"
server_key_iv: "28e72262dc18872f559e07da8b4cad58"
server_key_version: "encryptionkey@0"
I20260929 15:58:51.282078 2536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 15:58:51.285575 2552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:51.286928 2536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 15:58:51.287132 2536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "d1fb9ac8dd804eda894764f99bf2a494"
format_stamp: "Formatted at 2026-09-29 15:58:51 on dist-test-slave-cq88"
server_key: "7405d35cae7d7d0dea648a23f08a1b5f"
server_key_iv: "28e72262dc18872f559e07da8b4cad58"
server_key_version: "encryptionkey@0"
I20260929 15:58:51.287415 2536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:51.302732 2536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:51.307308 2536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:51.307624 2536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:51.308652 2536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:51.310099 2536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:51.310212 2536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:51.310361 2536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:51.310433 2536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:51.344996 2536 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:37177
I20260929 15:58:51.345026 2665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:37177 every 8 connection(s)
I20260929 15:58:51.346457 2536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:58:51.351164 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 2536
I20260929 15:58:51.351389 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:58:51.351840 24441 external_mini_cluster.cc:1428] Setting key 5e2ff97684575727c04ea009daa03175
Sep 29 15:58:51 dist-test-slave-cq88 krb5kdc[2294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:51.368317 2532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.349116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:41359 (local address 127.23.222.126:39669)
0929 15:58:51.349317 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:51.349322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:51.350610 (+ 1288us) server_negotiation.cc:407] Connection header received
0929 15:58:51.352633 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:51.352639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:51.352730 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:51.352842 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:51.354735 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.355975 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.357652 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.358477 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.358631 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.362233 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:51.362239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:51.362242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:51.362275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:51.364995 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:51.365912 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:51.365916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:51.365918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:51.366054 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:51.366552 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:51.366560 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:51.366562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:51.366818 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:51.366927 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:51.367891 (+ 964us) server_negotiation.cc:299] Negotiation successful
0929 15:58:51.368080 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 15:58:51.369918 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.349669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:39669 (local address 127.23.222.66:41359)
0929 15:58:51.350432 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:51.350497 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:51.352315 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:51.353151 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:51.353161 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:51.353962 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:51.354563 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.354581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.356156 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.356160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.357481 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.357511 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.358732 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.358736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.358965 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.359755 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:51.359778 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:51.361989 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:51.365143 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:51.365155 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:51.365176 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:51.365797 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:51.366163 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:51.366171 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:51.366178 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:51.366438 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:51.366943 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:51.366954 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:51.367487 (+ 533us) client_negotiation.cc:769] Sending connection context
0929 15:58:51.367793 (+ 306us) client_negotiation.cc:240] Negotiation successful
0929 15:58:51.368407 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":278,"threads_started":1}
I20260929 15:58:51.371644 2666 heartbeater.cc:347] Connected to a master server at 127.23.222.126:39669
I20260929 15:58:51.371977 2666 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:51.372970 2666 heartbeater.cc:511] Master 127.23.222.126:39669 requested a full tablet report, sending...
I20260929 15:58:51.374758 2340 ts_manager.cc:194] Registered new tserver with Master: d1fb9ac8dd804eda894764f99bf2a494 (127.23.222.66:37177)
I20260929 15:58:51.376194 2340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:41359
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:58:51.408811 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:39669
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:34861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:58:51.744225 2671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:58:51.744563 2671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:58:51.744714 2671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:58:51.752439 2671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:58:51.752573 2671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:58:51.752689 2671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:58:51.762537 2671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:34861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:39669
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:58:51.763804 2671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:58:51.765056 2671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:58:51.778280 2679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:51.778282 2676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:58:51.778564 2677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:58:51.779201 2671 server_base.cc:1034] running on GCE node
I20260929 15:58:51.779884 2671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:58:51.780956 2671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:58:51.782169 2671 hybrid_clock.cc:648] HybridClock initialized: now 1790697531782121 us; error 32 us; skew 500 ppm
I20260929 15:58:51.782573 2671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:58:51 dist-test-slave-cq88 krb5kdc[2294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:58:51.785986 2671 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:58:51.787653 2671 webserver.cc:460] Webserver started at http://127.23.222.67:39145/ using document root <none> and password file <none>
I20260929 15:58:51.788350 2671 fs_manager.cc:359] Metadata directory not provided
I20260929 15:58:51.788450 2671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:58:51.788695 2671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 15:58:51.791751 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "0ea6373f221f4cf8bb5c0f8657b90bab"
format_stamp: "Formatted at 2026-09-29 15:58:51 on dist-test-slave-cq88"
server_key: "684a066e331f545c9135954932419726"
server_key_iv: "5a9d64dd9a19ed8e964088c9ef77b431"
server_key_version: "encryptionkey@0"
I20260929 15:58:51.792459 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "0ea6373f221f4cf8bb5c0f8657b90bab"
format_stamp: "Formatted at 2026-09-29 15:58:51 on dist-test-slave-cq88"
server_key: "684a066e331f545c9135954932419726"
server_key_iv: "5a9d64dd9a19ed8e964088c9ef77b431"
server_key_version: "encryptionkey@0"
I20260929 15:58:51.797643 2671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 15:58:51.801414 2686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:51.802845 2671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 15:58:51.803022 2671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "0ea6373f221f4cf8bb5c0f8657b90bab"
format_stamp: "Formatted at 2026-09-29 15:58:51 on dist-test-slave-cq88"
server_key: "684a066e331f545c9135954932419726"
server_key_iv: "5a9d64dd9a19ed8e964088c9ef77b431"
server_key_version: "encryptionkey@0"
I20260929 15:58:51.803246 2671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:58:51.828984 2671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:58:51.833524 2671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:58:51.833837 2671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:58:51.834954 2671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:58:51.836349 2671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:58:51.836457 2671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:51.836609 2671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:58:51.836687 2671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:58:51.864949 2530 heartbeater.cc:503] Master 127.23.222.126:39669 was elected leader, sending a full tablet report...
I20260929 15:58:51.871397 2671 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:45141
I20260929 15:58:51.871452 2800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:45141 every 8 connection(s)
I20260929 15:58:51.872798 2671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:58:51.882089 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 2671
I20260929 15:58:51.882305 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:58:51.882761 24441 external_mini_cluster.cc:1428] Setting key 42602c4419357e76bb1fbf63186bbd0c
Sep 29 15:58:51 dist-test-slave-cq88 krb5kdc[2294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697531, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:58:51.893687 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.875550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:33697 (local address 127.23.222.126:39669)
0929 15:58:51.876209 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:51.876214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:51.877175 (+ 961us) server_negotiation.cc:407] Connection header received
0929 15:58:51.879038 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:51.879044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:51.879139 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:51.879281 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:51.880865 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.881673 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.883488 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.884049 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.884178 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.887680 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:51.887685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:51.887688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:51.887720 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:51.890335 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:51.891196 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:51.891201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:51.891203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:51.891391 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:51.891838 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:51.891843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:51.891847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:51.892140 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:51.892299 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:51.893071 (+ 772us) server_negotiation.cc:299] Negotiation successful
0929 15:58:51.893299 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":431,"threads_started":1}
I20260929 15:58:51.894665 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.876261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:39669 (local address 127.23.222.67:33697)
0929 15:58:51.877015 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:51.877059 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:51.878725 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:51.879548 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:51.879557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:51.880086 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:58:51.880687 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.880705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.881829 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.881841 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.883337 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.883350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.884284 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.884291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.884528 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.885230 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:58:51.885252 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:58:51.887461 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:58:51.890501 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:51.890510 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:51.890530 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:51.891060 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:51.891510 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:58:51.891515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:58:51.891519 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:58:51.891718 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:58:51.892277 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:58:51.892285 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:58:51.892686 (+ 401us) client_negotiation.cc:769] Sending connection context
0929 15:58:51.892962 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 15:58:51.893532 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":301,"threads_started":1}
I20260929 15:58:51.896100 2801 heartbeater.cc:347] Connected to a master server at 127.23.222.126:39669
I20260929 15:58:51.896441 2801 heartbeater.cc:464] Registering TS with master...
I20260929 15:58:51.897357 2801 heartbeater.cc:511] Master 127.23.222.126:39669 requested a full tablet report, sending...
I20260929 15:58:51.898877 2340 ts_manager.cc:194] Registered new tserver with Master: 0ea6373f221f4cf8bb5c0f8657b90bab (127.23.222.67:45141)
I20260929 15:58:51.899725 2340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:33697
I20260929 15:58:51.910729 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:58:51.928053 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.915343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.23.222.126:39669)
0929 15:58:51.915511 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:51.915515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:51.915978 (+ 463us) server_negotiation.cc:407] Connection header received
0929 15:58:51.916184 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:51.916191 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:51.916273 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:51.916379 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:51.917784 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.918511 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.919829 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.920271 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.920407 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.922666 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:51.922671 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:51.922673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:51.922705 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:51.925288 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:51.926006 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:51.926012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:51.926017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:51.926137 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:51.926510 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:51.926516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:51.926520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:51.926864 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:51.927117 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:51.927531 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 15:58:51.927743 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 15:58:51.932331 2340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:58:51.939661 2340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:58:51.966732 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.952918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:39099 (local address 127.0.0.1:44634)
0929 15:58:51.956459 (+ 3541us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:51.956520 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:51.956692 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:51.957880 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:51.957890 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:51.957929 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:51.958398 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.958414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.960811 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.960819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.963674 (+ 2855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.963691 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.965741 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.965747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.965956 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.966012 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 15:58:51.966151 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 15:58:51.966352 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3280,"thread_start_us":10348,"threads_started":1}
I20260929 15:58:51.966746 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.953328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.23.222.65:39099)
0929 15:58:51.954132 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:51.954144 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:51.956900 (+ 2756us) server_negotiation.cc:407] Connection header received
0929 15:58:51.957011 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:51.957021 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:51.957332 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:51.957552 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:51.958663 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.960633 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.963910 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.965529 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.965668 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.965925 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:51.966237 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 15:58:51.966421 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":377,"threads_started":1}
I20260929 15:58:51.971918 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.956638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.23.222.66:37177)
0929 15:58:51.963185 (+ 6547us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:51.963198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:51.964236 (+ 1038us) server_negotiation.cc:407] Connection header received
0929 15:58:51.964357 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:51.964366 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:51.964671 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:51.964887 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:51.965744 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.967283 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.969738 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.970794 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.970963 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.971174 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:51.971402 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 15:58:51.971560 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6320,"thread_start_us":6132,"threads_started":1}
I20260929 15:58:51.972353 2814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.953218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:45141 (local address 127.0.0.1:54196)
0929 15:58:51.963409 (+ 10191us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:51.963435 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:51.963562 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:51.964919 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:51.964925 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:51.964953 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:51.965482 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.965499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.967660 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.967670 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.969751 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.969765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.971548 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.971555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.971727 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.971776 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 15:58:51.971845 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 15:58:51.971968 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10059,"mutex_wait_us":4,"spinlock_wait_cycles":11008,"thread_start_us":2348,"threads_started":1}
I20260929 15:58:51.971714 2464 tablet_service.cc:1467] Processing CreateTablet for tablet 6065567357e54edf80d4eeba91dc8e6e (DEFAULT_TABLE table=test-table [id=04a7b3bc64e941a793c43b10bc031230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:51.972782 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.956945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:37177 (local address 127.0.0.1:54632)
0929 15:58:51.963970 (+ 7025us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:51.963989 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:51.964108 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:51.965075 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:51.965086 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:51.965134 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:51.965522 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.965536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.967492 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.967497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.969531 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:51.969545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.971015 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.971022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:51.971192 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.971232 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 15:58:51.971349 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 15:58:51.971506 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6908,"spinlock_wait_cycles":1792,"thread_start_us":6766,"threads_started":1}
I20260929 15:58:51.973737 2464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6065567357e54edf80d4eeba91dc8e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:51.973802 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:51.953522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.23.222.67:45141)
0929 15:58:51.964320 (+ 10798us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:51.964329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:51.964360 (+ 31us) server_negotiation.cc:407] Connection header received
0929 15:58:51.964460 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:51.964468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:51.964693 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:51.965690 (+ 997us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:51.965960 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.967524 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.969936 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:51.971307 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:51.972940 (+ 1633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:51.973214 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:51.973291 (+ 77us) server_negotiation.cc:299] Negotiation successful
0929 15:58:51.973459 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10623,"thread_start_us":11261,"threads_started":1}
I20260929 15:58:51.976608 2600 tablet_service.cc:1467] Processing CreateTablet for tablet 6065567357e54edf80d4eeba91dc8e6e (DEFAULT_TABLE table=test-table [id=04a7b3bc64e941a793c43b10bc031230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:51.977707 2600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6065567357e54edf80d4eeba91dc8e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:51.977342 2734 tablet_service.cc:1467] Processing CreateTablet for tablet 6065567357e54edf80d4eeba91dc8e6e (DEFAULT_TABLE table=test-table [id=04a7b3bc64e941a793c43b10bc031230]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:58:51.978717 2734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6065567357e54edf80d4eeba91dc8e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:58:51.988622 2820 tablet_bootstrap.cc:492] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: Bootstrap starting.
I20260929 15:58:51.992408 2820 tablet_bootstrap.cc:654] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:51.993564 2820 log.cc:824] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:51.994755 2821 tablet_bootstrap.cc:492] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: Bootstrap starting.
I20260929 15:58:51.996313 2824 tablet_bootstrap.cc:492] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: Bootstrap starting.
I20260929 15:58:51.996923 2820 tablet_bootstrap.cc:492] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: No bootstrap required, opened a new log
I20260929 15:58:51.997224 2820 ts_tablet_manager.cc:1397] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 15:58:51.997997 2821 tablet_bootstrap.cc:654] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:51.999012 2821 log.cc:824] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:51.999857 2824 tablet_bootstrap.cc:654] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: Neither blocks nor log segments found. Creating new log.
I20260929 15:58:52.001094 2824 log.cc:824] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: Log is configured to *not* fsync() on all Append() calls
I20260929 15:58:52.001950 2821 tablet_bootstrap.cc:492] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: No bootstrap required, opened a new log
I20260929 15:58:52.002329 2821 ts_tablet_manager.cc:1397] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 15:58:52.004776 2824 tablet_bootstrap.cc:492] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: No bootstrap required, opened a new log
I20260929 15:58:52.005107 2824 ts_tablet_manager.cc:1397] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260929 15:58:52.006254 2820 raft_consensus.cc:348] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } }
I20260929 15:58:52.006703 2820 raft_consensus.cc:374] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:52.006814 2820 raft_consensus.cc:729] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1fb9ac8dd804eda894764f99bf2a494, State: Initialized, Role: FOLLOWER
I20260929 15:58:52.007520 2820 consensus_queue.cc:260] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } }
I20260929 15:58:52.008708 2666 heartbeater.cc:503] Master 127.23.222.126:39669 was elected leader, sending a full tablet report...
I20260929 15:58:52.009287 2820 ts_tablet_manager.cc:1428] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 15:58:52.013819 2821 raft_consensus.cc:348] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } }
I20260929 15:58:52.014222 2821 raft_consensus.cc:374] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:52.014348 2821 raft_consensus.cc:729] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0af629e89a54d459357cb130eadf5b9, State: Initialized, Role: FOLLOWER
I20260929 15:58:52.014678 2824 raft_consensus.cc:348] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } }
I20260929 15:58:52.015051 2824 raft_consensus.cc:374] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:58:52.015117 2824 raft_consensus.cc:729] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea6373f221f4cf8bb5c0f8657b90bab, State: Initialized, Role: FOLLOWER
I20260929 15:58:52.015046 2821 consensus_queue.cc:260] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } }
I20260929 15:58:52.015775 2824 consensus_queue.cc:260] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } }
I20260929 15:58:52.017515 2821 ts_tablet_manager.cc:1428] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260929 15:58:52.018430 2801 heartbeater.cc:503] Master 127.23.222.126:39669 was elected leader, sending a full tablet report...
I20260929 15:58:52.019155 2824 ts_tablet_manager.cc:1428] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
W20260929 15:58:52.083985 2531 tablet.cc:2369] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:58:52.098631 2667 tablet.cc:2369] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:58:52.124495 2802 tablet.cc:2369] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:58:52.242363 2827 raft_consensus.cc:482] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:58:52.242719 2827 raft_consensus.cc:504] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } }
I20260929 15:58:52.244725 2827 leader_election.cc:288] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0af629e89a54d459357cb130eadf5b9 (127.23.222.65:39099), 0ea6373f221f4cf8bb5c0f8657b90bab (127.23.222.67:45141)
I20260929 15:58:52.252092 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.245208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:59205 (local address 127.23.222.65:39099)
0929 15:58:52.245446 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:52.245452 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:52.246285 (+ 833us) server_negotiation.cc:407] Connection header received
0929 15:58:52.246453 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:52.246460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:52.246551 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:52.246664 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:52.247321 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.248237 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.250179 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.251074 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.251283 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.251365 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:52.251712 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 15:58:52.251835 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 15:58:52.252226 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.245319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:38293 (local address 127.23.222.67:45141)
0929 15:58:52.245534 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:52.245543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:52.246417 (+ 874us) server_negotiation.cc:407] Connection header received
0929 15:58:52.246584 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:52.246589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:52.246663 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:52.246754 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:58:52.247557 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.248448 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.250343 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.251290 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.251480 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.251561 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:52.251854 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 15:58:52.251980 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 15:58:52.252327 2831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.245233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:45141 (local address 127.23.222.66:38293)
0929 15:58:52.246290 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:52.246310 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:52.246444 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:52.246857 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:52.246863 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:52.246891 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:52.247411 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.247420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.248589 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.248594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.250168 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.250180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.251506 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.251512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.251698 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.251720 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 15:58:52.251818 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 15:58:52.251937 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":40,"thread_start_us":620,"threads_started":1}
I20260929 15:58:52.252651 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.245164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:39099 (local address 127.23.222.66:59205)
0929 15:58:52.246158 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0929 15:58:52.246174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:58:52.246300 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:58:52.246677 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:58:52.246681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:58:52.246704 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:58:52.247144 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.247156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.248368 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.248377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.250028 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:58:52.250040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.251332 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.251339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:58:52.251504 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.251575 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 15:58:52.251686 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 15:58:52.251815 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":39,"thread_start_us":589,"threads_started":1}
I20260929 15:58:52.253535 2754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6065567357e54edf80d4eeba91dc8e6e" candidate_uuid: "d1fb9ac8dd804eda894764f99bf2a494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" is_pre_election: true
I20260929 15:58:52.253608 2485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6065567357e54edf80d4eeba91dc8e6e" candidate_uuid: "d1fb9ac8dd804eda894764f99bf2a494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0af629e89a54d459357cb130eadf5b9" is_pre_election: true
I20260929 15:58:52.254063 2754 raft_consensus.cc:2454] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1fb9ac8dd804eda894764f99bf2a494 in term 0.
I20260929 15:58:52.254113 2485 raft_consensus.cc:2454] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1fb9ac8dd804eda894764f99bf2a494 in term 0.
I20260929 15:58:52.254725 2556 leader_election.cc:302] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ea6373f221f4cf8bb5c0f8657b90bab, d0af629e89a54d459357cb130eadf5b9, d1fb9ac8dd804eda894764f99bf2a494; no voters:
I20260929 15:58:52.255095 2827 raft_consensus.cc:2781] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:58:52.255282 2827 raft_consensus.cc:482] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:58:52.255403 2827 raft_consensus.cc:3037] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:52.257524 2827 raft_consensus.cc:504] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } }
I20260929 15:58:52.258234 2827 leader_election.cc:288] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [CANDIDATE]: Term 1 election: Requested vote from peers d0af629e89a54d459357cb130eadf5b9 (127.23.222.65:39099), 0ea6373f221f4cf8bb5c0f8657b90bab (127.23.222.67:45141)
I20260929 15:58:52.258680 2485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6065567357e54edf80d4eeba91dc8e6e" candidate_uuid: "d1fb9ac8dd804eda894764f99bf2a494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0af629e89a54d459357cb130eadf5b9"
I20260929 15:58:52.258761 2754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6065567357e54edf80d4eeba91dc8e6e" candidate_uuid: "d1fb9ac8dd804eda894764f99bf2a494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab"
I20260929 15:58:52.258877 2485 raft_consensus.cc:3037] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:52.258999 2754 raft_consensus.cc:3037] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:58:52.261085 2485 raft_consensus.cc:2454] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1fb9ac8dd804eda894764f99bf2a494 in term 1.
I20260929 15:58:52.261266 2754 raft_consensus.cc:2454] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1fb9ac8dd804eda894764f99bf2a494 in term 1.
I20260929 15:58:52.261463 2555 leader_election.cc:302] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0af629e89a54d459357cb130eadf5b9, d1fb9ac8dd804eda894764f99bf2a494; no voters:
I20260929 15:58:52.261736 2827 raft_consensus.cc:2781] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:58:52.262115 2827 raft_consensus.cc:686] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 1 LEADER]: Becoming Leader. State: Replica: d1fb9ac8dd804eda894764f99bf2a494, State: Running, Role: LEADER
I20260929 15:58:52.262583 2827 consensus_queue.cc:237] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } }
I20260929 15:58:52.267408 2340 catalog_manager.cc:5495] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1fb9ac8dd804eda894764f99bf2a494 (127.23.222.66). New cstate: current_term: 1 leader_uuid: "d1fb9ac8dd804eda894764f99bf2a494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1fb9ac8dd804eda894764f99bf2a494" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37177 } health_report { overall_health: HEALTHY } } }
I20260929 15:58:52.331004 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.325497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.23.222.66:37177)
0929 15:58:52.325738 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:52.325746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:52.325848 (+ 102us) server_negotiation.cc:407] Connection header received
0929 15:58:52.326110 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:52.326114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:52.326197 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:52.326346 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:58:52.326724 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.327534 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.328830 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.329250 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.329383 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.329878 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:58:52.330403 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:58:52.330580 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:52.330633 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 15:58:52.330715 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:58:52.339210 2754 raft_consensus.cc:1261] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [term 1 FOLLOWER]: Refusing update from remote peer d1fb9ac8dd804eda894764f99bf2a494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:52.339365 2485 raft_consensus.cc:1261] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [term 1 FOLLOWER]: Refusing update from remote peer d1fb9ac8dd804eda894764f99bf2a494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:58:52.340339 2827 consensus_queue.cc:1035] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea6373f221f4cf8bb5c0f8657b90bab" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:52.340646 2832 consensus_queue.cc:1035] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0af629e89a54d459357cb130eadf5b9" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 39099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:58:52.383066 2340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:52.383389 2340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:58:52.387157 2340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4910c65495da44b6be90408c8f6efbc9: Sending DeleteTablet for 3 replicas of tablet 6065567357e54edf80d4eeba91dc8e6e
I20260929 15:58:52.388453 2464 tablet_service.cc:1514] Processing DeleteTablet for tablet 6065567357e54edf80d4eeba91dc8e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:52 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:44634
I20260929 15:58:52.388574 2734 tablet_service.cc:1514] Processing DeleteTablet for tablet 6065567357e54edf80d4eeba91dc8e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:52 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:54196
I20260929 15:58:52.389349 2845 tablet_replica.cc:330] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: stopping tablet replica
I20260929 15:58:52.389272 2600 tablet_service.cc:1514] Processing DeleteTablet for tablet 6065567357e54edf80d4eeba91dc8e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:58:52 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:54632
I20260929 15:58:52.389802 2845 raft_consensus.cc:2229] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:52.390252 2847 tablet_replica.cc:330] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: stopping tablet replica
I20260929 15:58:52.390385 2845 raft_consensus.cc:2258] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:52.390822 2847 raft_consensus.cc:2229] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:58:52.391635 2847 raft_consensus.cc:2258] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:52.392905 2845 ts_tablet_manager.cc:1905] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:52.393713 2847 ts_tablet_manager.cc:1905] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:58:52.393815 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 2401
I20260929 15:58:52.400430 2845 ts_tablet_manager.cc:1918] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:52.400669 2845 log.cc:1196] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/6065567357e54edf80d4eeba91dc8e6e
I20260929 15:58:52.401467 2845 ts_tablet_manager.cc:1939] T 6065567357e54edf80d4eeba91dc8e6e P d0af629e89a54d459357cb130eadf5b9: Deleting consensus metadata
I20260929 15:58:52.401471 2847 ts_tablet_manager.cc:1918] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:52.401791 2847 log.cc:1196] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/6065567357e54edf80d4eeba91dc8e6e
I20260929 15:58:52.402393 2847 ts_tablet_manager.cc:1939] T 6065567357e54edf80d4eeba91dc8e6e P d1fb9ac8dd804eda894764f99bf2a494: Deleting consensus metadata
I20260929 15:58:52.403611 2326 catalog_manager.cc:4841] TS d0af629e89a54d459357cb130eadf5b9 (127.23.222.65:39099): tablet 6065567357e54edf80d4eeba91dc8e6e (table test-table [id=04a7b3bc64e941a793c43b10bc031230]) successfully deleted
I20260929 15:58:52.404259 2327 catalog_manager.cc:4841] TS d1fb9ac8dd804eda894764f99bf2a494 (127.23.222.66:37177): tablet 6065567357e54edf80d4eeba91dc8e6e (table test-table [id=04a7b3bc64e941a793c43b10bc031230]) successfully deleted
I20260929 15:58:52.407823 2846 tablet_replica.cc:330] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: stopping tablet replica
I20260929 15:58:52.408335 2846 raft_consensus.cc:2229] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:58:52.408869 2846 raft_consensus.cc:2258] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:58:52.410708 2846 ts_tablet_manager.cc:1905] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 15:58:52 dist-test-slave-cq88 krb5kdc[2294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:58:52.417627 2846 ts_tablet_manager.cc:1918] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:58:52.417804 2846 log.cc:1196] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/6065567357e54edf80d4eeba91dc8e6e
I20260929 15:58:52.418226 2846 ts_tablet_manager.cc:1939] T 6065567357e54edf80d4eeba91dc8e6e P 0ea6373f221f4cf8bb5c0f8657b90bab: Deleting consensus metadata
I20260929 15:58:52.420158 2327 catalog_manager.cc:4841] TS 0ea6373f221f4cf8bb5c0f8657b90bab (127.23.222.67:45141): tablet 6065567357e54edf80d4eeba91dc8e6e (table test-table [id=04a7b3bc64e941a793c43b10bc031230]) successfully deleted
I20260929 15:58:52.425326 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.395504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40472 (local address 127.23.222.65:39099)
0929 15:58:52.395695 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:52.395701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:52.407457 (+ 11756us) server_negotiation.cc:407] Connection header received
0929 15:58:52.407753 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:52.407760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:52.407843 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:52.407944 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:52.409655 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.410790 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.412733 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.413384 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.413520 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.418884 (+ 5364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:52.418903 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:52.418922 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:52.418980 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:52.422140 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:52.423031 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:52.423038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:52.423040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:52.423184 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:52.423868 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:52.423872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:52.423874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:52.424254 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:52.424524 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:52.424828 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 15:58:52.424998 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 15:58:52.426249 2525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:52.516265 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 2401
I20260929 15:58:52.535142 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 2536
Sep 29 15:58:52 dist-test-slave-cq88 krb5kdc[2294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:58:52.552361 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.536061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.23.222.66:37177)
0929 15:58:52.536271 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:52.536277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:52.536297 (+ 20us) server_negotiation.cc:407] Connection header received
0929 15:58:52.536367 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:52.536373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:52.536454 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:52.536576 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:52.537696 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.538968 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.540484 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.541166 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.541292 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.545529 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:52.545546 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:52.545565 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:52.545612 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:52.549435 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:52.550088 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:52.550098 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:52.550101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:52.550272 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:52.550677 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:52.550683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:52.550686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:52.551181 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:52.551534 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:52.551678 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 15:58:52.551893 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 15:58:52.553117 2661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:52.667142 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 2536
I20260929 15:58:52.685810 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 2671
Sep 29 15:58:52 dist-test-slave-cq88 krb5kdc[2294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:58:52.700145 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:58:52.686517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.23.222.67:45141)
0929 15:58:52.686674 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 15:58:52.686678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:58:52.686716 (+ 38us) server_negotiation.cc:407] Connection header received
0929 15:58:52.686878 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:58:52.686882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:58:52.686943 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:58:52.687029 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:58:52.688370 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.689069 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.690230 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:58:52.690620 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:58:52.690710 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:58:52.695017 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:58:52.695032 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:58:52.695047 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:58:52.695088 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:58:52.697646 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:52.698269 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:52.698275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:52.698278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:52.698403 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:58:52.698703 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:58:52.698706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:58:52.698708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:58:52.699068 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:58:52.699354 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 15:58:52.699589 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 15:58:52.699778 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 15:58:52.700775 2796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:52.790490 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 2671
I20260929 15:58:52.807391 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 2307
I20260929 15:58:52.808550 2370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:58:52.917783 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 2307
2026-09-29T15:58:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4580 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:58:53 dist-test-slave-cq88 krb5kdc[2855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:58:53 dist-test-slave-cq88 krb5kdc[2855](info): set up 2 sockets
Sep 29 15:58:53 dist-test-slave-cq88 krb5kdc[2855](info): commencing operation
krb5kdc: starting...
W20260929 15:58:54.027145 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:58:54 dist-test-slave-cq88 krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:58:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/rangeradmin_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/rangeradmin_127.23.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/rangerlookup_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/rangerlookup_127.23.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/HTTP_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/HTTP_127.23.222.84@KRBTEST.COM.keytab.
I20260929 15:58:54.372146 24441 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGayxzf/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/postgres -l logfile start
2026-09-29 15:58:57.085 UTC [2885] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 15:58:57.085 UTC [2885] LOG: listening on IPv4 address "127.23.222.84", port 47321
2026-09-29 15:58:57.089 UTC [2885] LOG: listening on Unix socket "/tmp/.s.PGSQL.47321"
2026-09-29 15:58:57.111 UTC [2888] LOG: database system was shut down at 2026-09-29 15:58:55 UTC
2026-09-29 15:58:57.116 UTC [2885] LOG: database system is ready to accept connections
I20260929 15:58:58.090759 24441 mini_postgres.cc:96] Postgres bound to 47321
2026-09-29 15:58:58.109 UTC [2896] FATAL: database "slave" does not exist
127.23.222.84:47321 - accepting connections
I20260929 15:58:58.110242 24441 mini_ranger.cc:161] Starting Ranger...
I20260929 15:58:58.138051 24441 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 15:58:58.674407 24441 mini_ranger.cc:87] Created ranger Postgres database
I20260929 15:58:58.674526 24441 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-admin
2026-09-29 15:58:58,975 [I] DB FLAVOR :POSTGRES
2026-09-29 15:58:58,976 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:58:58,976 [I] Checking connection..
2026-09-29 15:58:58,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 15:58:59,235 [I] Checking connection passed.
2026-09-29 15:58:59,235 [I] --------- Verifying version history table ---------
2026-09-29 15:58:59,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:58:59,486 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:58:59,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:58:59,756 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:58:59,757 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 15:58:59,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 15:58:59.997 UTC [2961] WARNING: there is no transaction in progress
2026-09-29 15:59:00,004 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 15:59:00,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:59:00,277 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 15:59:00,277 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:59:00,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 15:59:00,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 15:59:00,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cq88.c.gcp-upstream.internal','N') ;"
2026-09-29 15:59:01,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:59:01,294 [I] Table x_portal_user does not exist in database ranger
2026-09-29 15:59:01,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:59:01,563 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 15:59:01,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:59:01,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:59:02,084 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 15:59:02,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 15:59:03.093 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.110 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.126 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.142 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.161 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.179 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.240 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.247 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.262 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.272 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.283 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.298 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.308 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.315 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.323 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:03.331 UTC [3070] WARNING: there is no transaction in progress
2026-09-29 15:59:04,012 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 15:59:04,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 15:59:04,260 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 15:59:04,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:59:04,497 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 15:59:04,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:59:04,768 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 15:59:04,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:59:04,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:59:05,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cq88.c.gcp-upstream.internal';"
2026-09-29 15:59:05,515 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 15:59:05,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 15:59:05,782 [I] DB_PATCHES have already been applied
I20260929 15:59:05.788460 24441 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGayxzf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 15:59:05.788599 24441 mini_ranger.cc:193] Using host: 127.23.222.84
I20260929 15:59:05.804177 24441 mini_ranger.cc:227] Ranger admin URL: http://127.23.222.84:41213
Sep 29, 2026 3:59:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:59:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:59:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:59:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:59:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:59:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:59:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 15:59:06.958889 24441 mini_ranger.cc:160] Time spent starting Ranger: real 8.849s user 0.004s sys 0.037s
Sep 29, 2026 3:59:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 3:59:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 29, 2026 3:59:27 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 3:59:27 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 3:59:28 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 15:59:28.892225 24441 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/rangerkms_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/rangerkms_127.23.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.222.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.222.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.222.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/HTTP_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.222.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/HTTP_127.23.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 15:59:29 dist-test-slave-cq88 krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697569, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 15:59:29.101999 24441 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 15:59:29.126789 24441 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 15:59:30.148490 24441 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 15:59:30.148684 24441 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-kms
I20260929 15:59:30.148828 24441 mini_ranger_kms.cc:227] Using postgres at 127.23.222.84:47321
2026-09-29 15:59:30,617 [I] DB FLAVOR :POSTGRES
2026-09-29 15:59:30,618 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:59:30,618 [I] Checking connection
2026-09-29 15:59:30,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 15:59:30,870 [I] connection success
2026-09-29 15:59:30,871 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 15:59:30,871 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 15:59:30,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 15:59:31,131 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 15:59:31,131 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:59:31,132 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 15:59:31,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:47321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 15:59:31,424 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 15:59:31.935105 24441 mini_ranger_kms.cc:310] Created kms service
I20260929 15:59:32.134483 24441 mini_ranger_kms.cc:326] Created kudu user
I20260929 15:59:32.184558 24441 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 15:59:32 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697569, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 15:59:32.635872 24441 mini_ranger_kms.cc:384] Added ranger policy
I20260929 15:59:32.636195 24441 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 15:59:32.636248 24441 mini_ranger_kms.cc:240] Using host: 127.23.222.84
I20260929 15:59:32.655694 24441 mini_ranger_kms.cc:276] Ranger KMS PID: 3252
I20260929 15:59:32.655944 24441 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.222.84:50825
Sep 29, 2026 3:59:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:59:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:59:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 3:59:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 3:59:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 3:59:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 15:59:34.980947 24441 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.879s user 0.111s sys 0.387s
I20260929 15:59:34.981215 24441 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 15:59:34.981478 24441 mini_ranger_kms.cc:401] 127.23.222.84:50825/kms/v1/keys
Sep 29 15:59:35 dist-test-slave-cq88 krb5kdc[2855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697575, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.222.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 15:59:35 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697575, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.222.84@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
Sep 29 15:59:36 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697575, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.222.84@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
Sep 29 15:59:38 dist-test-slave-cq88 krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 15:59:38.545428 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:41881
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:41881
--ranger_config_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:50825/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 15:59:38.901772 3303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:38.902173 3303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:38.902349 3303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:38.910596 3303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 15:59:38.910810 3303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:38.910928 3303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:38.911036 3303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 15:59:38.911123 3303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 15:59:38.922206 3303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.222.84:50825/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:41881
--ranger_config_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:41881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:59:38.923707 3303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:38.925277 3303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 15:59:38.939294 3303 server_base.cc:1034] running on GCE node
W20260929 15:59:38.939162 3308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:38.939122 3309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:38.939088 3311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:38.940721 3303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:38.942425 3303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:38.943759 3303 hybrid_clock.cc:648] HybridClock initialized: now 1790697578943651 us; error 34 us; skew 500 ppm
I20260929 15:59:38.944178 3303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:59:38 dist-test-slave-cq88 krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:38.951745 3303 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 15:59:38.953657 3303 webserver.cc:460] Webserver started at http://127.23.222.126:40143/ using document root <none> and password file <none>
I20260929 15:59:38.954403 3303 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:38.954555 3303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:38.954885 3303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:59:38 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697578, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 15:59:39.014529 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad"
format_stamp: "Formatted at 2026-09-29 15:59:39 on dist-test-slave-cq88"
server_key: "33d8ff13e791b870025cd0147de8f78d"
server_key_iv: "bcb3015f486c3dfa77bd0f6b5cd89b51"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:39.015601 3303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad"
format_stamp: "Formatted at 2026-09-29 15:59:39 on dist-test-slave-cq88"
server_key: "33d8ff13e791b870025cd0147de8f78d"
server_key_iv: "bcb3015f486c3dfa77bd0f6b5cd89b51"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:39.021786 3303 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 15:59:39.039836 3318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:39.041608 3303 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 15:59:39.041865 3303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad"
format_stamp: "Formatted at 2026-09-29 15:59:39 on dist-test-slave-cq88"
server_key: "33d8ff13e791b870025cd0147de8f78d"
server_key_iv: "bcb3015f486c3dfa77bd0f6b5cd89b51"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:39.042116 3303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:39.065716 3303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:39.070621 3303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:39.070986 3303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:39.093755 3303 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:41881
I20260929 15:59:39.093777 3369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:41881 every 8 connection(s)
I20260929 15:59:39.095191 3303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 15:59:39.098809 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 3303
I20260929 15:59:39.099064 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 15:59:39.099668 3370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:59:39.099949 24441 external_mini_cluster.cc:1428] Setting key 19f2d539cdbb925a2876fa3e57c2dda7
I20260929 15:59:39.115630 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad: Bootstrap starting.
Sep 29 15:59:39 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:59:39.119391 3370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:39.120600 3370 log.cc:824] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:39.123941 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad: No bootstrap required, opened a new log
I20260929 15:59:39.124917 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:39.102145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.23.222.126:41881)
0929 15:59:39.103007 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:39.103019 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:39.103073 (+ 54us) server_negotiation.cc:407] Connection header received
0929 15:59:39.104724 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:39.104747 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:39.105163 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:39.105514 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:39.107230 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:39.108777 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:39.110757 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:39.111721 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:39.111923 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:39.117012 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:39.117022 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:39.117039 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:39.117088 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:39.120393 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:39.121308 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:39.121316 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:39.121324 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:39.121508 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:39.121951 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:39.121956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:39.121960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:39.122522 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:39.122861 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:39.123381 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 15:59:39.123804 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":405,"threads_started":1}
I20260929 15:59:39.132720 3370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41881 } }
I20260929 15:59:39.133275 3370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:39.133404 3370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034dcc8ef8094d188be2a5b2e7dcb4ad, State: Initialized, Role: FOLLOWER
I20260929 15:59:39.134097 3370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41881 } }
I20260929 15:59:39.134343 3370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 15:59:39.134501 3370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 15:59:39.134691 3370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:39.137221 3370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41881 } }
I20260929 15:59:39.137789 3370 leader_election.cc:302] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 034dcc8ef8094d188be2a5b2e7dcb4ad; no voters:
I20260929 15:59:39.138484 3370 leader_election.cc:288] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 15:59:39.138656 3375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:59:39.139451 3375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [term 1 LEADER]: Becoming Leader. State: Replica: 034dcc8ef8094d188be2a5b2e7dcb4ad, State: Running, Role: LEADER
I20260929 15:59:39.140051 3375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41881 } }
I20260929 15:59:39.140844 3370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 15:59:39.143690 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 034dcc8ef8094d188be2a5b2e7dcb4ad. Latest consensus state: current_term: 1 leader_uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41881 } } }
I20260929 15:59:39.144178 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [sys.catalog]: This master's current role is: LEADER
I20260929 15:59:39.143839 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034dcc8ef8094d188be2a5b2e7dcb4ad" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41881 } } }
I20260929 15:59:39.144351 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad [sys.catalog]: This master's current role is: LEADER
I20260929 15:59:39.146723 3382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 15:59:39.153380 3382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 15:59:39.154201 3370 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 15:59:39.171851 3382 catalog_manager.cc:1347] Generated new cluster ID: ec2e296581634e438681823d10bc5be5
I20260929 15:59:39.172039 3382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 15:59:39.191072 3382 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 15:59:39.192571 3382 catalog_manager.cc:1504] Loading token signing keys...
I20260929 15:59:39.211078 3382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad: Generated new TSK 0
I20260929 15:59:39.212425 3382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 15:59:39.992480 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:50825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:41881
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:59:40 dist-test-slave-cq88 krb5kdc[2855](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 15:59:40.436422 3416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:40.436779 3416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:40.436939 3416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:40.445783 3416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:40.445960 3416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:40.446156 3416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 15:59:40.458633 3416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.222.84:50825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:41881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:59:40.460369 3416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:40.461968 3416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:40.479813 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:40.480049 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:40.480522 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:40.481520 3416 server_base.cc:1034] running on GCE node
I20260929 15:59:40.482364 3416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:40.483834 3416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:40.485136 3416 hybrid_clock.cc:648] HybridClock initialized: now 1790697580485072 us; error 54 us; skew 500 ppm
I20260929 15:59:40.485596 3416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:59:40 dist-test-slave-cq88 krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:40.490384 3416 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 15:59:40.492589 3416 webserver.cc:460] Webserver started at http://127.23.222.65:40263/ using document root <none> and password file <none>
I20260929 15:59:40.493646 3416 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:40.493814 3416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:40.494196 3416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:59:40 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 15:59:40.514793 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "2b0f03e134974e9e8dd5b06a68766ee2"
format_stamp: "Formatted at 2026-09-29 15:59:40 on dist-test-slave-cq88"
server_key: "5354645a381b276675e49298ed9197e8"
server_key_iv: "7892e1fccd99170cc72c93f70135b307"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:40.516216 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "2b0f03e134974e9e8dd5b06a68766ee2"
format_stamp: "Formatted at 2026-09-29 15:59:40 on dist-test-slave-cq88"
server_key: "5354645a381b276675e49298ed9197e8"
server_key_iv: "7892e1fccd99170cc72c93f70135b307"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:40.537781 3416 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.006s sys 0.013s
I20260929 15:59:40.567785 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:40.569732 3416 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.005s
I20260929 15:59:40.569969 3416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "2b0f03e134974e9e8dd5b06a68766ee2"
format_stamp: "Formatted at 2026-09-29 15:59:40 on dist-test-slave-cq88"
server_key: "5354645a381b276675e49298ed9197e8"
server_key_iv: "7892e1fccd99170cc72c93f70135b307"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:40.570237 3416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:40.593369 3416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:40.598172 3416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:40.598541 3416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:40.599822 3416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:40.601521 3416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:40.601701 3416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 15:59:40.601914 3416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:40.602054 3416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:40.662005 3416 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:40283
I20260929 15:59:40.662108 3548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:40283 every 8 connection(s)
I20260929 15:59:40.663412 3416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 15:59:40.670758 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 3416
I20260929 15:59:40.670997 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 15:59:40.672124 24441 external_mini_cluster.cc:1428] Setting key 797e4e7012310d4c5fceb8b2c7bbbdc2
Sep 29 15:59:40 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:59:40.701269 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:40.666467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:56611 (local address 127.23.222.126:41881)
0929 15:59:40.673925 (+ 7458us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:40.673935 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:40.674998 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 15:59:40.677374 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:40.677379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:40.677474 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:40.677617 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:40.679166 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:40.679985 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:40.681632 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:40.682045 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:40.682188 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:40.686539 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:40.686547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:40.686552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:40.686605 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:40.696732 (+ 10127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:40.697987 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:40.697995 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:40.697998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:40.698181 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:40.698859 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:40.698865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:40.698869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:40.699286 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:40.699421 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:40.700594 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0929 15:59:40.700852 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7269,"thread_start_us":7473,"threads_started":1}
I20260929 15:59:40.704090 3552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:40.666954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:41881 (local address 127.23.222.65:56611)
0929 15:59:40.674806 (+ 7852us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:40.674870 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:40.677087 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:40.677888 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:40.677897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:40.678364 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:40.678974 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:40.678993 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:40.680165 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:40.680169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:40.681475 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:40.681488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:40.682302 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:40.682309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:40.682558 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:40.683403 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:59:40.683434 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:40.686242 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:40.697009 (+ 10767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:40.697023 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:40.697048 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:40.697801 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:40.698358 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:40.698364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:40.698368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:40.698710 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:40.699578 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:40.699588 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:40.700097 (+ 509us) client_negotiation.cc:769] Sending connection context
0929 15:59:40.701802 (+ 1705us) client_negotiation.cc:240] Negotiation successful
0929 15:59:40.702471 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7502,"thread_start_us":7281,"threads_started":1}
I20260929 15:59:40.706415 3549 heartbeater.cc:347] Connected to a master server at 127.23.222.126:41881
I20260929 15:59:40.706847 3549 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:40.707870 3549 heartbeater.cc:511] Master 127.23.222.126:41881 requested a full tablet report, sending...
I20260929 15:59:40.711681 3335 ts_manager.cc:194] Registered new tserver with Master: 2b0f03e134974e9e8dd5b06a68766ee2 (127.23.222.65:40283)
I20260929 15:59:40.713836 3335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:56611
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 15:59:40.767495 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:50825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:41881
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 15:59:40 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
Sep 29 15:59:41 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697580, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
W20260929 15:59:41.282881 3555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:41.283420 3555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:41.283592 3555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:41.295341 3555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:41.295593 3555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:41.295841 3555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 15:59:41.311928 3555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.222.84:50825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:41881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:59:41.314026 3555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:41.315977 3555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:41.340937 3562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:41.349756 3563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:41.360687 3566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:41.362188 3555 server_base.cc:1034] running on GCE node
I20260929 15:59:41.363327 3555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:41.370975 3555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:41.372375 3555 hybrid_clock.cc:648] HybridClock initialized: now 1790697581372247 us; error 127 us; skew 500 ppm
I20260929 15:59:41.372864 3555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:59:41 dist-test-slave-cq88 krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697581, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:41.377396 3555 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 15:59:41.385394 3555 webserver.cc:460] Webserver started at http://127.23.222.66:44703/ using document root <none> and password file <none>
I20260929 15:59:41.386488 3555 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:41.386631 3555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:41.387020 3555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:59:41 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697581, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 15:59:41.406106 3555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "3cbeb1908a9e4b37ba592bf026aa8924"
format_stamp: "Formatted at 2026-09-29 15:59:41 on dist-test-slave-cq88"
server_key: "75da46d593cdec2f79c171063e4d1d1a"
server_key_iv: "e6c0ed6afc665acd160f5bf8d40cd81a"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:41.407369 3555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "3cbeb1908a9e4b37ba592bf026aa8924"
format_stamp: "Formatted at 2026-09-29 15:59:41 on dist-test-slave-cq88"
server_key: "75da46d593cdec2f79c171063e4d1d1a"
server_key_iv: "e6c0ed6afc665acd160f5bf8d40cd81a"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:41.421648 3555 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.007s sys 0.005s
I20260929 15:59:41.444096 3577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:41.446138 3555 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260929 15:59:41.446442 3555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "3cbeb1908a9e4b37ba592bf026aa8924"
format_stamp: "Formatted at 2026-09-29 15:59:41 on dist-test-slave-cq88"
server_key: "75da46d593cdec2f79c171063e4d1d1a"
server_key_iv: "e6c0ed6afc665acd160f5bf8d40cd81a"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:41.446792 3555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:41.479492 3555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:41.485076 3555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:41.485462 3555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:41.486760 3555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:41.488755 3555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:41.488869 3555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:41.489005 3555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:41.489073 3555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:41.538471 3555 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:34615
I20260929 15:59:41.538523 3694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:34615 every 8 connection(s)
I20260929 15:59:41.540172 3555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 15:59:41.549624 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 3555
I20260929 15:59:41.549824 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 15:59:41.550266 24441 external_mini_cluster.cc:1428] Setting key 5ff06cffb9e7c60553eb5b2c14673730
Sep 29 15:59:41 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697581, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:59:41.568634 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:41.543315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:51199 (local address 127.23.222.126:41881)
0929 15:59:41.543883 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:41.543890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:41.545159 (+ 1269us) server_negotiation.cc:407] Connection header received
0929 15:59:41.547823 (+ 2664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:41.547827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:41.547900 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:41.548020 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:41.550094 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:41.551417 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:41.557374 (+ 5957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:41.558107 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:41.558261 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:41.562279 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:41.562284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:41.562288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:41.562323 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:41.565161 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:41.566015 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:41.566021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:41.566025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:41.566162 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:41.566658 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:41.566667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:41.566670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:41.566966 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:41.567128 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:41.567972 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 15:59:41.568231 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":292,"threads_started":1}
I20260929 15:59:41.569747 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:41.544103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:41881 (local address 127.23.222.66:51199)
0929 15:59:41.544934 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:41.545003 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:41.547445 (+ 2442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:41.548317 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:41.548330 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:41.548999 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:41.549857 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:41.549883 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:41.551609 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:41.551616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:41.557122 (+ 5506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:41.557148 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:41.558413 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:41.558424 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:41.558783 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:41.559730 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:59:41.559755 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:41.562009 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:41.565354 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:41.565365 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:41.565386 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:41.565862 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:41.566291 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:41.566296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:41.566300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:41.566522 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:41.567108 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:41.567118 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:41.567571 (+ 453us) client_negotiation.cc:769] Sending connection context
0929 15:59:41.567852 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 15:59:41.568436 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":307,"threads_started":1}
I20260929 15:59:41.571475 3695 heartbeater.cc:347] Connected to a master server at 127.23.222.126:41881
I20260929 15:59:41.572021 3695 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:41.572904 3695 heartbeater.cc:511] Master 127.23.222.126:41881 requested a full tablet report, sending...
I20260929 15:59:41.574738 3335 ts_manager.cc:194] Registered new tserver with Master: 3cbeb1908a9e4b37ba592bf026aa8924 (127.23.222.66:34615)
I20260929 15:59:41.575742 3335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:51199
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 15:59:41.617067 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:50825/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:41881
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 15:59:41.717958 3549 heartbeater.cc:503] Master 127.23.222.126:41881 was elected leader, sending a full tablet report...
W20260929 15:59:41.989684 3701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 15:59:41.990032 3701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 15:59:41.990212 3701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 15:59:41.997797 3701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 15:59:41.997993 3701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 15:59:41.998176 3701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 15:59:42.008533 3701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.222.84:50825/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:41881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 15:59:42.009966 3701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 15:59:42.011415 3701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 15:59:42.025563 3707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:42.025580 3709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 15:59:42.025668 3706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 15:59:42.026448 3701 server_base.cc:1034] running on GCE node
I20260929 15:59:42.027149 3701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 15:59:42.028465 3701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 15:59:42.029713 3701 hybrid_clock.cc:648] HybridClock initialized: now 1790697582029634 us; error 69 us; skew 500 ppm
I20260929 15:59:42.030120 3701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 15:59:42 dist-test-slave-cq88 krb5kdc[2855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697582, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 15:59:42.033913 3701 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 15:59:42.035856 3701 webserver.cc:460] Webserver started at http://127.23.222.67:46503/ using document root <none> and password file <none>
I20260929 15:59:42.036651 3701 fs_manager.cc:359] Metadata directory not provided
I20260929 15:59:42.036819 3701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 15:59:42.037159 3701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 15:59:42 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697582, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 15:59:42.055518 3701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "0a04cbf8715b451398b0c0fd51bc2b90"
format_stamp: "Formatted at 2026-09-29 15:59:42 on dist-test-slave-cq88"
server_key: "af6b3f07e9d1fa638226968638407919"
server_key_iv: "2b05de384070b9da3a286ac4996f3a8e"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:42.056703 3701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "0a04cbf8715b451398b0c0fd51bc2b90"
format_stamp: "Formatted at 2026-09-29 15:59:42 on dist-test-slave-cq88"
server_key: "af6b3f07e9d1fa638226968638407919"
server_key_iv: "2b05de384070b9da3a286ac4996f3a8e"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:42.064507 3701 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 15:59:42.081373 3716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:42.082886 3701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 15:59:42.083110 3701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "0a04cbf8715b451398b0c0fd51bc2b90"
format_stamp: "Formatted at 2026-09-29 15:59:42 on dist-test-slave-cq88"
server_key: "af6b3f07e9d1fa638226968638407919"
server_key_iv: "2b05de384070b9da3a286ac4996f3a8e"
server_key_version: "kuduclusterkey@0"
I20260929 15:59:42.083446 3701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 15:59:42.115739 3701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 15:59:42.120064 3701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 15:59:42.120397 3701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 15:59:42.121456 3701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 15:59:42.123078 3701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 15:59:42.123183 3701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:42.123346 3701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 15:59:42.123418 3701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 15:59:42.160614 3701 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:36763
I20260929 15:59:42.160658 3829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:36763 every 8 connection(s)
I20260929 15:59:42.161952 3701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 15:59:42.166553 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 3701
I20260929 15:59:42.166724 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 15:59:42.167124 24441 external_mini_cluster.cc:1428] Setting key 8541152dc3fbd049a80cbcac126a5333
Sep 29 15:59:42 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697582, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 15:59:42.185016 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.164723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:58431 (local address 127.23.222.126:41881)
0929 15:59:42.165256 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.165263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.166337 (+ 1074us) server_negotiation.cc:407] Connection header received
0929 15:59:42.168417 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.168433 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.168560 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.168690 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:42.170286 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.171068 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.172852 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.173343 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.173527 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.177169 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:42.177173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:42.177177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:42.177210 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:42.180960 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.181995 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.182001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.182004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.182168 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.182812 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.182817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.182820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.183202 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:42.183377 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.184388 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.184621 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":237,"threads_started":1}
I20260929 15:59:42.186091 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.165393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:41881 (local address 127.23.222.67:58431)
0929 15:59:42.166169 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:42.166210 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:42.168127 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:42.168948 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:42.168958 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:42.169425 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0929 15:59:42.170066 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.170085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.171276 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.171285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.172672 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.172685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.173633 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.173641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.173913 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.174706 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 15:59:42.174736 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 15:59:42.176944 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 15:59:42.181198 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:42.181208 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:42.181229 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:42.181836 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:42.182316 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 15:59:42.182321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 15:59:42.182325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 15:59:42.182669 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 15:59:42.183400 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 15:59:42.183407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 15:59:42.183935 (+ 528us) client_negotiation.cc:769] Sending connection context
0929 15:59:42.184292 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 15:59:42.184853 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":283,"threads_started":1}
I20260929 15:59:42.187624 3830 heartbeater.cc:347] Connected to a master server at 127.23.222.126:41881
I20260929 15:59:42.188032 3830 heartbeater.cc:464] Registering TS with master...
I20260929 15:59:42.188892 3830 heartbeater.cc:511] Master 127.23.222.126:41881 requested a full tablet report, sending...
I20260929 15:59:42.190414 3335 ts_manager.cc:194] Registered new tserver with Master: 0a04cbf8715b451398b0c0fd51bc2b90 (127.23.222.67:36763)
I20260929 15:59:42.191287 3335 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:58431
I20260929 15:59:42.195698 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 15:59:42.215281 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.200316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.23.222.126:41881)
0929 15:59:42.200511 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.200516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.201033 (+ 517us) server_negotiation.cc:407] Connection header received
0929 15:59:42.201210 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.201214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.201303 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.201396 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:42.202900 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.203916 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.205340 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.206275 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.206446 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.209522 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:42.209530 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:42.209535 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:42.209586 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:42.212241 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.213028 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.213035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.213039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.213179 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.213633 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.213640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.213644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.213961 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:42.214243 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.214636 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.214875 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 15:59:42.219980 3335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 15:59:42.227372 3335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 15:59:42.256628 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.241909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.23.222.67:36763)
0929 15:59:42.242574 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.242583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.244210 (+ 1627us) server_negotiation.cc:407] Connection header received
0929 15:59:42.244708 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.244721 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.245040 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.245242 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:42.246633 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.248838 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.253183 (+ 4345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.254858 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.254961 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.255195 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.256056 (+ 861us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.256242 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":1051,"threads_started":1}
I20260929 15:59:42.257565 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.241516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:36763 (local address 127.0.0.1:43300)
0929 15:59:42.244065 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:42.244108 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:42.244379 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:42.245767 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:42.245778 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:42.245819 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:42.246354 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.246368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.249164 (+ 2796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.249173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.252915 (+ 3742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.252943 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.255466 (+ 2523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.255472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.255719 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.255781 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 15:59:42.256931 (+ 1150us) client_negotiation.cc:240] Negotiation successful
0929 15:59:42.257142 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2320,"thread_start_us":15799,"threads_started":1}
I20260929 15:59:42.262157 3765 tablet_service.cc:1467] Processing CreateTablet for tablet d8c53f1a5a784c79a44228e107bf1abc (DEFAULT_TABLE table=test-table [id=813cd023cd654326aeb658c8dd01afb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:59:42.263357 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.242787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.23.222.66:34615)
0929 15:59:42.243323 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.243331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.254250 (+ 10919us) server_negotiation.cc:407] Connection header received
0929 15:59:42.254800 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.254810 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.255134 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.255348 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:42.256317 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.257836 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.260347 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.261753 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.261888 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.262095 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.262775 (+ 680us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.262953 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":364,"threads_started":1}
I20260929 15:59:42.263801 3765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c53f1a5a784c79a44228e107bf1abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:59:42.264071 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.242327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:34615 (local address 127.0.0.1:52940)
0929 15:59:42.254118 (+ 11791us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:42.254140 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:42.254593 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:42.255715 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:42.255719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:42.255746 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:42.256150 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.256162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.257997 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.258001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.260174 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.260188 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.262286 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.262293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.262557 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.262596 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 15:59:42.263316 (+ 720us) client_negotiation.cc:240] Negotiation successful
0929 15:59:42.263677 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11634,"mutex_wait_us":33,"thread_start_us":11954,"threads_started":1}
I20260929 15:59:42.265664 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.241512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.23.222.65:40283)
0929 15:59:42.257089 (+ 15577us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.257103 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.258125 (+ 1022us) server_negotiation.cc:407] Connection header received
0929 15:59:42.258238 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.258249 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.258519 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.258743 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:42.259407 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.260734 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.262856 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.264567 (+ 1711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.264710 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.264928 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.265176 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.265331 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15359,"thread_start_us":16349,"threads_started":1}
I20260929 15:59:42.266719 3844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.241515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:40283 (local address 127.0.0.1:37340)
0929 15:59:42.257644 (+ 16129us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:42.257666 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:42.257782 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:42.258748 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:42.258754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:42.258784 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:42.259164 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.259176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.260880 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.260886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.262653 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.262667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.264799 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.264806 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.264977 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.265009 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 15:59:42.266352 (+ 1343us) client_negotiation.cc:240] Negotiation successful
0929 15:59:42.266457 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15970,"mutex_wait_us":1868,"thread_start_us":352,"threads_started":1}
I20260929 15:59:42.267935 3629 tablet_service.cc:1467] Processing CreateTablet for tablet d8c53f1a5a784c79a44228e107bf1abc (DEFAULT_TABLE table=test-table [id=813cd023cd654326aeb658c8dd01afb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:59:42.269439 3629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c53f1a5a784c79a44228e107bf1abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:59:42.270469 3484 tablet_service.cc:1467] Processing CreateTablet for tablet d8c53f1a5a784c79a44228e107bf1abc (DEFAULT_TABLE table=test-table [id=813cd023cd654326aeb658c8dd01afb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 15:59:42.272322 3484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c53f1a5a784c79a44228e107bf1abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 15:59:42.279311 3849 tablet_bootstrap.cc:492] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: Bootstrap starting.
I20260929 15:59:42.283493 3849 tablet_bootstrap.cc:654] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:42.284960 3849 log.cc:824] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:42.284991 3850 tablet_bootstrap.cc:492] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: Bootstrap starting.
I20260929 15:59:42.287673 3852 tablet_bootstrap.cc:492] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: Bootstrap starting.
I20260929 15:59:42.288669 3849 tablet_bootstrap.cc:492] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: No bootstrap required, opened a new log
I20260929 15:59:42.288879 3849 ts_tablet_manager.cc:1397] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 15:59:42.288924 3850 tablet_bootstrap.cc:654] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:42.290243 3850 log.cc:824] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:42.291560 3852 tablet_bootstrap.cc:654] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: Neither blocks nor log segments found. Creating new log.
I20260929 15:59:42.292698 3852 log.cc:824] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: Log is configured to *not* fsync() on all Append() calls
I20260929 15:59:42.294193 3850 tablet_bootstrap.cc:492] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: No bootstrap required, opened a new log
I20260929 15:59:42.294472 3850 ts_tablet_manager.cc:1397] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 15:59:42.295435 3852 tablet_bootstrap.cc:492] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: No bootstrap required, opened a new log
I20260929 15:59:42.295692 3852 ts_tablet_manager.cc:1397] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 15:59:42.299337 3849 raft_consensus.cc:348] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } }
I20260929 15:59:42.299823 3849 raft_consensus.cc:374] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:42.299917 3849 raft_consensus.cc:729] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a04cbf8715b451398b0c0fd51bc2b90, State: Initialized, Role: FOLLOWER
I20260929 15:59:42.300655 3849 consensus_queue.cc:260] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } }
I20260929 15:59:42.301819 3830 heartbeater.cc:503] Master 127.23.222.126:41881 was elected leader, sending a full tablet report...
I20260929 15:59:42.302604 3849 ts_tablet_manager.cc:1428] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 15:59:42.304534 3850 raft_consensus.cc:348] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } }
I20260929 15:59:42.304620 3852 raft_consensus.cc:348] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } }
I20260929 15:59:42.304930 3850 raft_consensus.cc:374] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:42.305045 3850 raft_consensus.cc:729] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cbeb1908a9e4b37ba592bf026aa8924, State: Initialized, Role: FOLLOWER
I20260929 15:59:42.305074 3852 raft_consensus.cc:374] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 15:59:42.305171 3852 raft_consensus.cc:729] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0f03e134974e9e8dd5b06a68766ee2, State: Initialized, Role: FOLLOWER
I20260929 15:59:42.305668 3850 consensus_queue.cc:260] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } }
I20260929 15:59:42.305879 3852 consensus_queue.cc:260] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } }
I20260929 15:59:42.306799 3695 heartbeater.cc:503] Master 127.23.222.126:41881 was elected leader, sending a full tablet report...
I20260929 15:59:42.307580 3850 ts_tablet_manager.cc:1428] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260929 15:59:42.309305 3852 ts_tablet_manager.cc:1428] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260929 15:59:42.413628 3831 tablet.cc:2369] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 15:59:42.416254 3550 tablet.cc:2369] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:59:42.473762 3856 raft_consensus.cc:482] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 15:59:42.474107 3856 raft_consensus.cc:504] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } }
I20260929 15:59:42.476046 3856 leader_election.cc:288] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b0f03e134974e9e8dd5b06a68766ee2 (127.23.222.65:40283), 0a04cbf8715b451398b0c0fd51bc2b90 (127.23.222.67:36763)
I20260929 15:59:42.485445 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.476486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:59623 (local address 127.23.222.65:40283)
0929 15:59:42.476740 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.476749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.477312 (+ 563us) server_negotiation.cc:407] Connection header received
0929 15:59:42.477562 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.477567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.477666 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.477779 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:42.478524 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.479696 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.482870 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.484174 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.484350 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.484426 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.485072 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.485195 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 15:59:42.485437 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.476596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:39465 (local address 127.23.222.67:36763)
0929 15:59:42.476978 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.476989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.477626 (+ 637us) server_negotiation.cc:407] Connection header received
0929 15:59:42.477773 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.477778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.477848 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.477936 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 15:59:42.478661 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.479696 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.482866 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.484149 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.484350 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.484426 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.485054 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.485186 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260929 15:59:42.485714 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.476265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:40283 (local address 127.23.222.66:59623)
0929 15:59:42.477194 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:42.477210 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:42.477406 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:42.477806 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:42.477809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:42.477826 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:42.478316 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.478336 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.479851 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.479857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.482692 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.482709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.484567 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.484576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.484811 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.484895 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 15:59:42.484985 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 15:59:42.485148 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":219,"thread_start_us":575,"threads_started":1}
I20260929 15:59:42.485739 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.476508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:36763 (local address 127.23.222.66:39465)
0929 15:59:42.477521 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0929 15:59:42.477537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 15:59:42.477652 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 15:59:42.478036 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 15:59:42.478040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 15:59:42.478061 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 15:59:42.478507 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.478521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.479852 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.479857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.482693 (+ 2836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 15:59:42.482709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.484575 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.484582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 15:59:42.484813 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.484872 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 15:59:42.485018 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 15:59:42.485149 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":282,"thread_start_us":686,"threads_started":1}
I20260929 15:59:42.486847 3785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c53f1a5a784c79a44228e107bf1abc" candidate_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" is_pre_election: true
I20260929 15:59:42.486967 3504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c53f1a5a784c79a44228e107bf1abc" candidate_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" is_pre_election: true
I20260929 15:59:42.487483 3785 raft_consensus.cc:2454] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbeb1908a9e4b37ba592bf026aa8924 in term 0.
I20260929 15:59:42.487483 3504 raft_consensus.cc:2454] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cbeb1908a9e4b37ba592bf026aa8924 in term 0.
I20260929 15:59:42.488245 3580 leader_election.cc:302] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a04cbf8715b451398b0c0fd51bc2b90, 2b0f03e134974e9e8dd5b06a68766ee2, 3cbeb1908a9e4b37ba592bf026aa8924; no voters:
I20260929 15:59:42.488696 3856 raft_consensus.cc:2781] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 15:59:42.488858 3856 raft_consensus.cc:482] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 15:59:42.488976 3856 raft_consensus.cc:3037] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:42.491146 3856 raft_consensus.cc:504] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } }
I20260929 15:59:42.491882 3856 leader_election.cc:288] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [CANDIDATE]: Term 1 election: Requested vote from peers 2b0f03e134974e9e8dd5b06a68766ee2 (127.23.222.65:40283), 0a04cbf8715b451398b0c0fd51bc2b90 (127.23.222.67:36763)
I20260929 15:59:42.492403 3504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c53f1a5a784c79a44228e107bf1abc" candidate_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b0f03e134974e9e8dd5b06a68766ee2"
I20260929 15:59:42.492489 3785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c53f1a5a784c79a44228e107bf1abc" candidate_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a04cbf8715b451398b0c0fd51bc2b90"
I20260929 15:59:42.492654 3504 raft_consensus.cc:3037] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:42.492702 3785 raft_consensus.cc:3037] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [term 0 FOLLOWER]: Advancing to term 1
I20260929 15:59:42.496074 3504 raft_consensus.cc:2454] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbeb1908a9e4b37ba592bf026aa8924 in term 1.
I20260929 15:59:42.496073 3785 raft_consensus.cc:2454] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cbeb1908a9e4b37ba592bf026aa8924 in term 1.
I20260929 15:59:42.496532 3580 leader_election.cc:302] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0a04cbf8715b451398b0c0fd51bc2b90, 2b0f03e134974e9e8dd5b06a68766ee2, 3cbeb1908a9e4b37ba592bf026aa8924; no voters:
I20260929 15:59:42.496855 3856 raft_consensus.cc:2781] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 15:59:42.497511 3856 raft_consensus.cc:686] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 1 LEADER]: Becoming Leader. State: Replica: 3cbeb1908a9e4b37ba592bf026aa8924, State: Running, Role: LEADER
I20260929 15:59:42.498085 3856 consensus_queue.cc:237] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } }
I20260929 15:59:42.503484 3334 catalog_manager.cc:5495] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cbeb1908a9e4b37ba592bf026aa8924 (127.23.222.66). New cstate: current_term: 1 leader_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cbeb1908a9e4b37ba592bf026aa8924" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 34615 } health_report { overall_health: HEALTHY } } }
W20260929 15:59:42.542742 3696 tablet.cc:2369] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 15:59:42.570261 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.564408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.23.222.66:34615)
0929 15:59:42.564640 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.564646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.564754 (+ 108us) server_negotiation.cc:407] Connection header received
0929 15:59:42.565039 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.565043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.565122 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.565300 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 15:59:42.565712 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.566448 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.568015 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.568425 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.568571 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.569108 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 15:59:42.569632 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 15:59:42.569813 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.569873 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.569960 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 15:59:42.579591 3785 raft_consensus.cc:1261] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [term 1 FOLLOWER]: Refusing update from remote peer 3cbeb1908a9e4b37ba592bf026aa8924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:59:42.579622 3504 raft_consensus.cc:1261] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [term 1 FOLLOWER]: Refusing update from remote peer 3cbeb1908a9e4b37ba592bf026aa8924: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 15:59:42.580866 3856 consensus_queue.cc:1035] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b0f03e134974e9e8dd5b06a68766ee2" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:59:42.581205 3860 consensus_queue.cc:1035] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a04cbf8715b451398b0c0fd51bc2b90" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 15:59:42.601163 3865 mvcc.cc:205] Tried to move back new op lower bound from 7334697298234458112 to 7334697297917382656. Current Snapshot: MvccSnapshot[applied={T|T < 7334697298234458112}]
I20260929 15:59:42.613283 3334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:59:42.613644 3334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 15:59:42.617802 3334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 034dcc8ef8094d188be2a5b2e7dcb4ad: Sending DeleteTablet for 3 replicas of tablet d8c53f1a5a784c79a44228e107bf1abc
I20260929 15:59:42.619105 3765 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c53f1a5a784c79a44228e107bf1abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:59:42 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:43300
I20260929 15:59:42.619138 3484 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c53f1a5a784c79a44228e107bf1abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:59:42 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:37340
I20260929 15:59:42.619499 3629 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c53f1a5a784c79a44228e107bf1abc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 15:59:42 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:52940
I20260929 15:59:42.622057 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 3416
I20260929 15:59:42.630472 3873 tablet_replica.cc:330] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: stopping tablet replica
I20260929 15:59:42.631274 3873 raft_consensus.cc:2229] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:59:42.632063 3873 raft_consensus.cc:2258] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:59:42.634862 3873 ts_tablet_manager.cc:1905] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:59:42.641641 3872 tablet_replica.cc:330] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: stopping tablet replica
I20260929 15:59:42.642315 3872 raft_consensus.cc:2229] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 15:59:42.642936 3872 raft_consensus.cc:2258] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:59:42.644963 3873 ts_tablet_manager.cc:1918] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:59:42.645224 3873 log.cc:1196] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/d8c53f1a5a784c79a44228e107bf1abc
I20260929 15:59:42.645902 3873 ts_tablet_manager.cc:1939] T d8c53f1a5a784c79a44228e107bf1abc P 2b0f03e134974e9e8dd5b06a68766ee2: Deleting consensus metadata
I20260929 15:59:42.646301 3872 ts_tablet_manager.cc:1905] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:59:42.648160 3322 catalog_manager.cc:4841] TS 2b0f03e134974e9e8dd5b06a68766ee2 (127.23.222.65:40283): tablet d8c53f1a5a784c79a44228e107bf1abc (table test-table [id=813cd023cd654326aeb658c8dd01afb7]) successfully deleted
I20260929 15:59:42.648588 3874 tablet_replica.cc:330] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: stopping tablet replica
I20260929 15:59:42.649051 3874 raft_consensus.cc:2229] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 1 LEADER]: Raft consensus shutting down.
I20260929 15:59:42.649822 3874 raft_consensus.cc:2258] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 15:59:42.652076 3874 ts_tablet_manager.cc:1905] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 15:59:42.655045 3872 ts_tablet_manager.cc:1918] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:59:42.655206 3872 log.cc:1196] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/d8c53f1a5a784c79a44228e107bf1abc
Sep 29 15:59:42 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 15:59:42.655725 3872 ts_tablet_manager.cc:1939] T d8c53f1a5a784c79a44228e107bf1abc P 0a04cbf8715b451398b0c0fd51bc2b90: Deleting consensus metadata
I20260929 15:59:42.657622 3319 catalog_manager.cc:4841] TS 0a04cbf8715b451398b0c0fd51bc2b90 (127.23.222.67:36763): tablet d8c53f1a5a784c79a44228e107bf1abc (table test-table [id=813cd023cd654326aeb658c8dd01afb7]) successfully deleted
I20260929 15:59:42.659578 3874 ts_tablet_manager.cc:1918] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 15:59:42.659775 3874 log.cc:1196] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/d8c53f1a5a784c79a44228e107bf1abc
I20260929 15:59:42.660250 3874 ts_tablet_manager.cc:1939] T d8c53f1a5a784c79a44228e107bf1abc P 3cbeb1908a9e4b37ba592bf026aa8924: Deleting consensus metadata
I20260929 15:59:42.661859 3320 catalog_manager.cc:4841] TS 3cbeb1908a9e4b37ba592bf026aa8924 (127.23.222.66:34615): tablet d8c53f1a5a784c79a44228e107bf1abc (table test-table [id=813cd023cd654326aeb658c8dd01afb7]) successfully deleted
I20260929 15:59:42.665185 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.624022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.23.222.65:40283)
0929 15:59:42.624262 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.624271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.643956 (+ 19685us) server_negotiation.cc:407] Connection header received
0929 15:59:42.644191 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.644197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.644283 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.644402 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:42.646740 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.647915 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.650351 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.651004 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.651113 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.657125 (+ 6012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:42.657147 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:42.657167 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:42.657217 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:42.661385 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.662468 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.662478 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.662482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.662646 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.663228 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.663260 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.663264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.663776 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:42.664074 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.664491 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.664765 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 15:59:42.665983 3544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:42.761492 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 3416
I20260929 15:59:42.785713 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 3555
Sep 29 15:59:42 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 15:59:42.806643 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.786665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.23.222.66:34615)
0929 15:59:42.786849 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.786853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.786872 (+ 19us) server_negotiation.cc:407] Connection header received
0929 15:59:42.787164 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.787169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.787310 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.787440 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:42.789250 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.790155 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.792280 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.792745 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.792856 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.798544 (+ 5688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:42.798568 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:42.798590 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:42.798647 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:42.803133 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.804111 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.804121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.804127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.804331 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.804836 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.804842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.804846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.805320 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:42.805652 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.805982 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.806225 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 15:59:42.807498 3690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:42.922343 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 3555
I20260929 15:59:42.937906 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 3701
Sep 29 15:59:42 dist-test-slave-cq88 krb5kdc[2855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 15:59:42.955479 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 15:59:42.938710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.23.222.67:36763)
0929 15:59:42.938933 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 15:59:42.938941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 15:59:42.939054 (+ 113us) server_negotiation.cc:407] Connection header received
0929 15:59:42.939322 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 15:59:42.939334 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 15:59:42.939450 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 15:59:42.939577 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 15:59:42.941129 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.941956 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.943413 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 15:59:42.943834 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 15:59:42.943985 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 15:59:42.948966 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 15:59:42.948983 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 15:59:42.948999 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 15:59:42.949036 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 15:59:42.952301 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.953105 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.953113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.953116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.953270 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 15:59:42.953733 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 15:59:42.953740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 15:59:42.953744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 15:59:42.954158 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 15:59:42.954500 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 15:59:42.954828 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 15:59:42.955065 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 15:59:42.956231 3825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:43.046835 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 3701
I20260929 15:59:43.063782 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 3303
I20260929 15:59:43.065138 3365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 15:59:43.177351 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 3303
I20260929 15:59:43.204597 24441 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 15:59:43.550927 24441 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 15:59:43.551142 24441 mini_ranger.cc:66] Stopping Ranger...
I20260929 15:59:43.897390 24441 mini_ranger.cc:68] Stopped Ranger
2026-09-29 15:59:43.897 UTC [2885] LOG: received smart shutdown request
2026-09-29 15:59:43.901 UTC [2885] LOG: background worker "logical replication launcher" (PID 2894) exited with exit code 1
2026-09-29 15:59:43.901 UTC [2889] LOG: shutting down
2026-09-29 15:59:44.101 UTC [2885] LOG: database system is shut down
2026-09-29T15:59:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51338 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 15:59:44 dist-test-slave-cq88 krb5kdc[3888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 15:59:44 dist-test-slave-cq88 krb5kdc[3888](info): set up 2 sockets
Sep 29 15:59:44 dist-test-slave-cq88 krb5kdc[3888](info): commencing operation
krb5kdc: starting...
W20260929 15:59:45.407195 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 15:59:45 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T15:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T15:59:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/rangeradmin_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/rangeradmin_127.23.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/rangerlookup_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/rangerlookup_127.23.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/HTTP_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/HTTP_127.23.222.84@KRBTEST.COM.keytab.
I20260929 15:59:45.759909 24441 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGayxzf/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/postgres -l logfile start
2026-09-29 15:59:48.950 UTC [3919] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 15:59:48.950 UTC [3919] LOG: listening on IPv4 address "127.23.222.84", port 57043
2026-09-29 15:59:48.954 UTC [3919] LOG: listening on Unix socket "/tmp/.s.PGSQL.57043"
2026-09-29 15:59:48.980 UTC [3922] LOG: database system was shut down at 2026-09-29 15:59:46 UTC
2026-09-29 15:59:48.986 UTC [3919] LOG: database system is ready to accept connections
I20260929 15:59:49.982560 24441 mini_postgres.cc:96] Postgres bound to 57043
2026-09-29 15:59:50.000 UTC [3930] FATAL: database "slave" does not exist
127.23.222.84:57043 - accepting connections
I20260929 15:59:50.001345 24441 mini_ranger.cc:161] Starting Ranger...
I20260929 15:59:50.029145 24441 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 15:59:50.611749 24441 mini_ranger.cc:87] Created ranger Postgres database
I20260929 15:59:50.611864 24441 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-admin
2026-09-29 15:59:50,891 [I] DB FLAVOR :POSTGRES
2026-09-29 15:59:50,891 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 15:59:50,892 [I] Checking connection..
2026-09-29 15:59:50,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 15:59:51,142 [I] Checking connection passed.
2026-09-29 15:59:51,142 [I] --------- Verifying version history table ---------
2026-09-29 15:59:51,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:59:51,402 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:59:51,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:59:51,656 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 15:59:51,656 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 15:59:51,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 15:59:51.909 UTC [3994] WARNING: there is no transaction in progress
2026-09-29 15:59:51,916 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 15:59:51,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 15:59:52,186 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 15:59:52,186 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 15:59:52,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 15:59:52,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 15:59:52,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cq88.c.gcp-upstream.internal','N') ;"
2026-09-29 15:59:52,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:59:53,231 [I] Table x_portal_user does not exist in database ranger
2026-09-29 15:59:53,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:59:53,488 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 15:59:53,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:59:53,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:59:53,992 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 15:59:53,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 15:59:55.067 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.084 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.102 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.119 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.137 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.158 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.217 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.224 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.239 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.251 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.262 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.276 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.287 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.294 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.302 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:55.309 UTC [4103] WARNING: there is no transaction in progress
2026-09-29 15:59:56,066 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 15:59:56,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 15:59:56,325 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 15:59:56,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 15:59:56,591 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 15:59:56,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 15:59:56,849 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 15:59:56,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 15:59:57,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 15:59:57,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cq88.c.gcp-upstream.internal';"
2026-09-29 15:59:57,639 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 15:59:57,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 15:59:57,877 [I] DB_PATCHES have already been applied
I20260929 15:59:57.883188 24441 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGayxzf/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 15:59:57.883334 24441 mini_ranger.cc:193] Using host: 127.23.222.84
I20260929 15:59:57.898048 24441 mini_ranger.cc:227] Ranger admin URL: http://127.23.222.84:56015
Sep 29, 2026 3:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 3:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 3:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 3:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 3:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 3:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 3:59:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 15:59:59.045540 24441 mini_ranger.cc:160] Time spent starting Ranger: real 9.044s user 0.005s sys 0.034s
Sep 29, 2026 4:00:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 4:00:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 4:00:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 4:00:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 4:00:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 16:00:21.817363 24441 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.222.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.222.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/rangerkms_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.222.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/rangerkms_127.23.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.222.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.222.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.222.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/HTTP_127.23.222.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.222.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/HTTP_127.23.222.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 16:00:21 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697621, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 16:00:22.000665 24441 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 16:00:22.025285 24441 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 16:00:23.036473 24441 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 16:00:23.036603 24441 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-kms
I20260929 16:00:23.036662 24441 mini_ranger_kms.cc:227] Using postgres at 127.23.222.84:57043
2026-09-29 16:00:23,404 [I] DB FLAVOR :POSTGRES
2026-09-29 16:00:23,405 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 16:00:23,405 [I] Checking connection
2026-09-29 16:00:23,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 16:00:23,667 [I] connection success
2026-09-29 16:00:23,667 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 16:00:23,667 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 16:00:23,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 16:00:23,900 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 16:00:23,900 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 16:00:23,900 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 16:00:23,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.222.84:57043/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 16:00:24,167 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 16:00:24.689916 24441 mini_ranger_kms.cc:310] Created kms service
I20260929 16:00:24.920050 24441 mini_ranger_kms.cc:326] Created kudu user
I20260929 16:00:24.966184 24441 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 16:00:24 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697621, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:25.388067 24441 mini_ranger_kms.cc:384] Added ranger policy
I20260929 16:00:25.388396 24441 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGayxzf/build/asan/bin/postgresql.jar:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGayxzf/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 16:00:25.388463 24441 mini_ranger_kms.cc:240] Using host: 127.23.222.84
I20260929 16:00:25.406654 24441 mini_ranger_kms.cc:276] Ranger KMS PID: 4285
I20260929 16:00:25.406801 24441 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.222.84:35125
Sep 29, 2026 4:00:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 4:00:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 4:00:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 4:00:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 4:00:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 4:00:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 16:00:27 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697627, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.222.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:27.746158 24441 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.745s user 0.027s sys 0.364s
I20260929 16:00:27.746399 24441 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 16:00:27.746575 24441 mini_ranger_kms.cc:401] 127.23.222.84:35125/kms/v1/keys
Sep 29 16:00:27 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697627, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.222.84@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
Sep 29 16:00:28 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697627, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.222.84@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
Sep 29 16:00:30 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:00:30.580693 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37461
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:37461
--ranger_config_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:35125/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:30.932905 4335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:30.933213 4335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:30.933385 4335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:30.940824 4335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:30.940948 4335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:30.941041 4335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:30.941149 4335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:30.941218 4335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:30.951713 4335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.222.84:35125/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:37461
--ranger_config_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:30.953068 4335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:30.954478 4335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:30.967061 4340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:30.967016 4343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:30.967089 4341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:30.967901 4335 server_base.cc:1034] running on GCE node
I20260929 16:00:30.968674 4335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:30.970069 4335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:30.971395 4335 hybrid_clock.cc:648] HybridClock initialized: now 1790697630971340 us; error 37 us; skew 500 ppm
I20260929 16:00:30.971738 4335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:30 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697630, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:30.975463 4335 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:00:30.976904 4335 webserver.cc:460] Webserver started at http://127.23.222.126:37941/ using document root <none> and password file <none>
I20260929 16:00:30.977532 4335 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:30.977623 4335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:30.977882 4335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:30 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697630, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:31.044356 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd"
format_stamp: "Formatted at 2026-09-29 16:00:31 on dist-test-slave-cq88"
server_key: "06e3c55892c6c1674225712d3c78f3ff"
server_key_iv: "cf116886b645e75021a9c80f6062d165"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:31.045259 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd"
format_stamp: "Formatted at 2026-09-29 16:00:31 on dist-test-slave-cq88"
server_key: "06e3c55892c6c1674225712d3c78f3ff"
server_key_iv: "cf116886b645e75021a9c80f6062d165"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:31.051097 4335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:00:31.071038 4351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:31.072834 4335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 16:00:31.073017 4335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd"
format_stamp: "Formatted at 2026-09-29 16:00:31 on dist-test-slave-cq88"
server_key: "06e3c55892c6c1674225712d3c78f3ff"
server_key_iv: "cf116886b645e75021a9c80f6062d165"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:31.073268 4335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:31.092619 4335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:31.097143 4335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:31.097471 4335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:31.117271 4335 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:37461
I20260929 16:00:31.117324 4402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:37461 every 8 connection(s)
I20260929 16:00:31.118659 4335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:00:31.121604 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 4335
I20260929 16:00:31.121824 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:00:31.122515 24441 external_mini_cluster.cc:1428] Setting key 2cc9ef72b8eceb4d680f5b071652d9d5
I20260929 16:00:31.123409 4403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:31.138527 4403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Bootstrap starting.
I20260929 16:00:31.143816 4403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:31.145232 4403 log.cc:824] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:31.148926 4403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: No bootstrap required, opened a new log
Sep 29 16:00:31 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:31.159725 4406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:31.124809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.23.222.126:37461)
0929 16:00:31.139841 (+ 15032us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:31.139855 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:31.139905 (+ 50us) server_negotiation.cc:407] Connection header received
0929 16:00:31.141700 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:31.141725 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:31.142162 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:31.142588 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:31.144194 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:31.145611 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:31.147354 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:31.148068 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:31.148271 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:31.152273 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:31.152281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:31.152296 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:31.152338 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:31.155430 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:31.156130 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:31.156136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:31.156143 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:31.156289 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:31.156688 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:31.156692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:31.156697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:31.157245 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:31.157591 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:31.158008 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 16:00:31.158480 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14785,"thread_start_us":14513,"threads_started":1}
I20260929 16:00:31.159833 4403 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } }
I20260929 16:00:31.160300 4403 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:31.160429 4403 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e7f58ee1ea4a20b46f54bbad4c83dd, State: Initialized, Role: FOLLOWER
I20260929 16:00:31.161235 4403 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } }
I20260929 16:00:31.161532 4403 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:00:31.161664 4403 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:00:31.161829 4403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:31.165169 4403 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } }
I20260929 16:00:31.165768 4403 leader_election.cc:302] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e7f58ee1ea4a20b46f54bbad4c83dd; no voters:
I20260929 16:00:31.166482 4403 leader_election.cc:288] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:00:31.166548 4408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:00:31.168802 4403 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:31.173115 4408 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 1 LEADER]: Becoming Leader. State: Replica: d2e7f58ee1ea4a20b46f54bbad4c83dd, State: Running, Role: LEADER
I20260929 16:00:31.173655 4408 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } }
I20260929 16:00:31.176512 4409 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } } }
I20260929 16:00:31.176956 4409 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:31.177484 4410 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e7f58ee1ea4a20b46f54bbad4c83dd. Latest consensus state: current_term: 1 leader_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } } }
I20260929 16:00:31.177763 4410 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:31.183533 4415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:00:31.189878 4415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:00:31.194680 4403 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 16:00:31.219957 4415 catalog_manager.cc:1347] Generated new cluster ID: 634cdde9f20941bbaeb7d9532726d1c3
I20260929 16:00:31.220114 4415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:00:31.229771 4415 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:00:31.231467 4415 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:00:31.239794 4415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Generated new TSK 0
I20260929 16:00:31.240731 4415 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 16:00:32.060611 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:35125/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37461
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:32 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697632, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 16:00:32.509799 4449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:32.510215 4449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:32.510432 4449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:32.522269 4449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:32.522413 4449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:32.522655 4449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:00:32.538709 4449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.222.84:35125/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:32.540642 4449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:32.542383 4449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:32.566417 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:32.567852 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:32.572827 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:32.573063 4449 server_base.cc:1034] running on GCE node
I20260929 16:00:32.574115 4449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:32.576532 4449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:32.577878 4449 hybrid_clock.cc:648] HybridClock initialized: now 1790697632577762 us; error 109 us; skew 500 ppm
I20260929 16:00:32.578322 4449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:32 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697632, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:32.582434 4449 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:00:32.585704 4449 webserver.cc:460] Webserver started at http://127.23.222.65:41939/ using document root <none> and password file <none>
I20260929 16:00:32.586503 4449 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:32.586660 4449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:32.586987 4449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:32 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697632, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:32.605803 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "757a837d0cef49a5a1e125aff38b118b"
format_stamp: "Formatted at 2026-09-29 16:00:32 on dist-test-slave-cq88"
server_key: "0983d691e27055058b80c6ef9bc50fc8"
server_key_iv: "ee2c20a92cbf988fcd2e1929d983f169"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:32.606734 4449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "757a837d0cef49a5a1e125aff38b118b"
format_stamp: "Formatted at 2026-09-29 16:00:32 on dist-test-slave-cq88"
server_key: "0983d691e27055058b80c6ef9bc50fc8"
server_key_iv: "ee2c20a92cbf988fcd2e1929d983f169"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:32.613250 4449 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 16:00:32.639562 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:32.641685 4449 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.002s
I20260929 16:00:32.641922 4449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "757a837d0cef49a5a1e125aff38b118b"
format_stamp: "Formatted at 2026-09-29 16:00:32 on dist-test-slave-cq88"
server_key: "0983d691e27055058b80c6ef9bc50fc8"
server_key_iv: "ee2c20a92cbf988fcd2e1929d983f169"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:32.642220 4449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:32.706828 4449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:32.712626 4449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:32.713086 4449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:32.714473 4449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:32.716604 4449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:32.716758 4449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:00:32.716980 4449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:32.717120 4449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:32.796773 4449 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:45037
I20260929 16:00:32.796795 4582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:45037 every 8 connection(s)
I20260929 16:00:32.798619 4449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:00:32.806905 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 4449
I20260929 16:00:32.807112 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 16:00:32.807937 24441 external_mini_cluster.cc:1428] Setting key 23a9fcbbc85a7f2fa1aaecc5b1ef25e2
Sep 29 16:00:32 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697632, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:32.836777 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:32.801603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:45707 (local address 127.23.222.126:37461)
0929 16:00:32.815294 (+ 13691us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:32.815301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:32.815322 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:00:32.815383 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:32.815388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:32.815472 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:32.815593 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:32.817892 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:32.819065 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:32.821640 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:32.822316 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:32.822444 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:32.827148 (+ 4704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:32.827156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:32.827161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:32.827208 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:32.830934 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:32.832060 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:32.832067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:32.832071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:32.832223 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:32.832850 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:32.832855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:32.832859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:32.833234 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:32.833375 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:32.836122 (+ 2747us) server_negotiation.cc:299] Negotiation successful
0929 16:00:32.836371 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13509,"thread_start_us":13367,"threads_started":1}
I20260929 16:00:32.838043 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:32.802486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37461 (local address 127.23.222.65:45707)
0929 16:00:32.811963 (+ 9477us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:32.812029 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:32.814453 (+ 2424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:32.816084 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:32.816098 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:32.816777 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:32.817657 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:32.817684 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:32.819292 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:32.819298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:32.821401 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:32.821420 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:32.822787 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:32.822794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:32.823116 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:32.824010 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:32.824043 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:32.826808 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:32.831183 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:32.831193 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:32.831215 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:32.831887 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:32.832360 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:32.832364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:32.832368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:32.832709 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:32.833457 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:32.833466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:32.833943 (+ 477us) client_negotiation.cc:769] Sending connection context
0929 16:00:32.834278 (+ 335us) client_negotiation.cc:240] Negotiation successful
0929 16:00:32.834854 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9142,"thread_start_us":8870,"threads_started":1}
I20260929 16:00:32.840106 4583 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37461
I20260929 16:00:32.840453 4583 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:32.841382 4583 heartbeater.cc:511] Master 127.23.222.126:37461 requested a full tablet report, sending...
I20260929 16:00:32.844208 4368 ts_manager.cc:194] Registered new tserver with Master: 757a837d0cef49a5a1e125aff38b118b (127.23.222.65:45037)
I20260929 16:00:32.846285 4368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:45707
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Sep 29 16:00:32 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697632, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 16:00:32.899883 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:35125/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37461
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:33 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697632, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
W20260929 16:00:33.450960 4589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:33.451510 4589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:33.451715 4589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:33.463035 4589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:33.463281 4589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:33.463538 4589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:00:33.479281 4589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.222.84:35125/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:33.481284 4589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:33.483088 4589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:33.523813 4597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:33.524910 4596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:33.525126 4599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:33.525660 4589 server_base.cc:1034] running on GCE node
I20260929 16:00:33.526695 4589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:33.535624 4589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:33.539199 4589 hybrid_clock.cc:648] HybridClock initialized: now 1790697633539058 us; error 134 us; skew 500 ppm
I20260929 16:00:33.539791 4589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:33 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697633, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:33.544351 4589 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:00:33.548058 4589 webserver.cc:460] Webserver started at http://127.23.222.66:40639/ using document root <none> and password file <none>
I20260929 16:00:33.548988 4589 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:33.549176 4589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:33.549597 4589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:33 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697633, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:33.570209 4589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "2d137b12f8a24d9382203509f0f05b2b"
format_stamp: "Formatted at 2026-09-29 16:00:33 on dist-test-slave-cq88"
server_key: "6342c777ef3df81ac66ce35004879aa9"
server_key_iv: "e5b088f384e11dadbe35074395a5724f"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:33.571455 4589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "2d137b12f8a24d9382203509f0f05b2b"
format_stamp: "Formatted at 2026-09-29 16:00:33 on dist-test-slave-cq88"
server_key: "6342c777ef3df81ac66ce35004879aa9"
server_key_iv: "e5b088f384e11dadbe35074395a5724f"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:33.579536 4589 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260929 16:00:33.606542 4609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:33.608449 4589 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 16:00:33.608692 4589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "2d137b12f8a24d9382203509f0f05b2b"
format_stamp: "Formatted at 2026-09-29 16:00:33 on dist-test-slave-cq88"
server_key: "6342c777ef3df81ac66ce35004879aa9"
server_key_iv: "e5b088f384e11dadbe35074395a5724f"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:33.608990 4589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:33.626883 4589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:33.630518 4589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:33.630774 4589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:33.631774 4589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:33.633116 4589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:33.633188 4589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:33.633270 4589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:33.633317 4589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:33.667101 4589 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:46287
I20260929 16:00:33.667150 4726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:46287 every 8 connection(s)
I20260929 16:00:33.668531 4589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:00:33.678614 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 4589
I20260929 16:00:33.678797 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 16:00:33.679383 24441 external_mini_cluster.cc:1428] Setting key 4968ed5dc517d230ec46c97a2eadb083
Sep 29 16:00:33 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697633, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:33.691753 4729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:33.671413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:55243 (local address 127.23.222.126:37461)
0929 16:00:33.671881 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:33.671886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:33.672879 (+ 993us) server_negotiation.cc:407] Connection header received
0929 16:00:33.674727 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:33.674732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:33.674837 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:33.674967 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:33.676485 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:33.677207 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:33.679298 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:33.680033 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:33.680233 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:33.684326 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:33.684339 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:33.684345 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:33.684390 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:33.687824 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:33.688808 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:33.688817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:33.688827 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:33.688976 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:33.689546 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:33.689557 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:33.689560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:33.689914 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:33.690099 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:33.690968 (+ 869us) server_negotiation.cc:299] Negotiation successful
0929 16:00:33.691266 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":236,"threads_started":1}
I20260929 16:00:33.692915 4730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:33.671937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37461 (local address 127.23.222.66:55243)
0929 16:00:33.672704 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:33.672753 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:33.674435 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:33.675222 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:33.675231 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:33.675724 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:33.676306 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:33.676324 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:33.677366 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:33.677371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:33.679061 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:33.679086 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:33.680366 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:33.680411 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:33.680680 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:33.681447 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:33.681475 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:33.684043 (+ 2568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:33.688018 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:33.688030 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:33.688052 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:33.688645 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:33.689117 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:33.689123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:33.689127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:33.689393 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:33.690074 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:33.690085 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:33.690506 (+ 421us) client_negotiation.cc:769] Sending connection context
0929 16:00:33.690833 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 16:00:33.691516 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":341,"threads_started":1}
I20260929 16:00:33.694708 4727 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37461
I20260929 16:00:33.695088 4727 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:33.696079 4727 heartbeater.cc:511] Master 127.23.222.126:37461 requested a full tablet report, sending...
I20260929 16:00:33.697896 4368 ts_manager.cc:194] Registered new tserver with Master: 2d137b12f8a24d9382203509f0f05b2b (127.23.222.66:46287)
I20260929 16:00:33.699347 4368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:55243
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 16:00:33.735888 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:35125/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37461
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:33.850473 4583 heartbeater.cc:503] Master 127.23.222.126:37461 was elected leader, sending a full tablet report...
W20260929 16:00:34.063781 4733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:34.064060 4733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:34.064235 4733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:34.071774 4733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:34.071902 4733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:34.072089 4733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:00:34.081966 4733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.222.84:35125/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:34.083319 4733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:34.084673 4733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:34.097983 4738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:34.098059 4739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:34.098088 4741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:34.099113 4733 server_base.cc:1034] running on GCE node
I20260929 16:00:34.099812 4733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:34.100880 4733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:34.102104 4733 hybrid_clock.cc:648] HybridClock initialized: now 1790697634102069 us; error 69 us; skew 500 ppm
I20260929 16:00:34.102433 4733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:34 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697634, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:34.105887 4733 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:00:34.107626 4733 webserver.cc:460] Webserver started at http://127.23.222.67:33427/ using document root <none> and password file <none>
I20260929 16:00:34.108348 4733 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:34.108497 4733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:34.108819 4733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 16:00:34 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697634, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:34.125047 4733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "7fbae2062b09427b89f2235dde034d55"
format_stamp: "Formatted at 2026-09-29 16:00:34 on dist-test-slave-cq88"
server_key: "505663573bb123012664265e749f5b51"
server_key_iv: "9268ea23c395a6c89c5bee10a8cf18b0"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:34.126094 4733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "7fbae2062b09427b89f2235dde034d55"
format_stamp: "Formatted at 2026-09-29 16:00:34 on dist-test-slave-cq88"
server_key: "505663573bb123012664265e749f5b51"
server_key_iv: "9268ea23c395a6c89c5bee10a8cf18b0"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:34.133849 4733 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 16:00:34.151520 4748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:34.153136 4733 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:00:34.153335 4733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "7fbae2062b09427b89f2235dde034d55"
format_stamp: "Formatted at 2026-09-29 16:00:34 on dist-test-slave-cq88"
server_key: "505663573bb123012664265e749f5b51"
server_key_iv: "9268ea23c395a6c89c5bee10a8cf18b0"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:34.153580 4733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:34.194736 4733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:34.199285 4733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:34.199616 4733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:34.200645 4733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:34.202111 4733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:34.202230 4733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:34.202376 4733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:34.202453 4733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:34.239146 4733 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:36543
I20260929 16:00:34.239274 4861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:36543 every 8 connection(s)
I20260929 16:00:34.240483 4733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:00:34.249280 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 4733
I20260929 16:00:34.249480 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 16:00:34.250006 24441 external_mini_cluster.cc:1428] Setting key 7a7c497d119b092b0c4e0c745eb5717b
Sep 29 16:00:34 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697634, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:34.265429 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:34.243474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:48533 (local address 127.23.222.126:37461)
0929 16:00:34.244006 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:34.244013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:34.245130 (+ 1117us) server_negotiation.cc:407] Connection header received
0929 16:00:34.246928 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:34.246934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:34.247083 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:34.247208 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:34.249238 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.250375 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.252910 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.253526 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.253673 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:34.258018 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:34.258025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:34.258030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:34.258101 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:34.261728 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:34.262651 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:34.262657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:34.262660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:34.262823 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:34.263318 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:34.263323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:34.263327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:34.263679 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:34.263837 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:34.264617 (+ 780us) server_negotiation.cc:299] Negotiation successful
0929 16:00:34.265063 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":253,"threads_started":1}
I20260929 16:00:34.266577 4865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:34.244128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37461 (local address 127.23.222.67:48533)
0929 16:00:34.244932 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:34.244984 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:34.246653 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:34.247481 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:34.247489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:34.247908 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:34.248922 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:34.248953 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.250582 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.250589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:34.252734 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:34.252759 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.253772 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.253782 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:34.254070 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:34.255382 (+ 1312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:34.255410 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:34.257678 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:34.261907 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:34.261916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:34.261931 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:34.262493 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:34.262936 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:34.262940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:34.262942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:34.263163 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:34.263800 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:34.263806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:34.264153 (+ 347us) client_negotiation.cc:769] Sending connection context
0929 16:00:34.264479 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 16:00:34.265023 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":350,"threads_started":1}
I20260929 16:00:34.268041 4862 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37461
I20260929 16:00:34.268392 4862 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:34.269357 4862 heartbeater.cc:511] Master 127.23.222.126:37461 requested a full tablet report, sending...
I20260929 16:00:34.271006 4368 ts_manager.cc:194] Registered new tserver with Master: 7fbae2062b09427b89f2235dde034d55 (127.23.222.67:36543)
I20260929 16:00:34.271937 4368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:48533
I20260929 16:00:34.278909 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:00:34.281469 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 4449
Sep 29 16:00:34 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 16:00:34.299427 4867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:34.282439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.23.222.65:45037)
0929 16:00:34.283270 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:34.283282 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:34.283316 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:00:34.283374 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:34.283381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:34.283673 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:34.283839 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:34.285464 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.286471 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.288128 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.288714 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.288858 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:34.293443 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:34.293455 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:34.293469 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:34.293505 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:34.296432 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:34.297187 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:34.297194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:34.297197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:34.297331 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:34.297739 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:34.297745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:34.297748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:34.298098 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:34.298438 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:34.298731 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:00:34.298997 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":418,"threads_started":1}
I20260929 16:00:34.300784 4578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:34.417320 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 4449
I20260929 16:00:34.434188 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 4589
Sep 29 16:00:34 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 16:00:34.453014 4869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:34.435185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53722 (local address 127.23.222.66:46287)
0929 16:00:34.435760 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:34.435767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:34.435792 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:00:34.435857 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:34.435863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:34.436066 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:34.436260 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:34.437611 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.438566 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.440109 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.440956 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.441123 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:34.446215 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:34.446229 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:34.446245 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:34.446285 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:34.449496 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:34.450584 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:34.450593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:34.450596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:34.450741 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:34.451205 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:34.451208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:34.451210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:34.451701 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:34.451926 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:34.452364 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 16:00:34.452608 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":242,"threads_started":1}
I20260929 16:00:34.454490 4722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:34.539927 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 4589
I20260929 16:00:34.560592 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 4733
Sep 29 16:00:34 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 16:00:34.577922 4871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:34.561660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.23.222.67:36543)
0929 16:00:34.562321 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:34.562329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:34.562359 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:00:34.562419 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:34.562426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:34.562648 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:34.562827 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:34.564383 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.565413 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.566858 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:34.567498 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:34.567642 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:34.572519 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:34.572531 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:34.572551 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:34.572586 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:34.575197 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:34.575908 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:34.575915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:34.575917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:34.576051 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:34.576505 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:34.576508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:34.576510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:34.576851 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:34.577106 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:34.577421 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 16:00:34.577624 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":352,"threads_started":1}
I20260929 16:00:34.579409 4857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:34.662588 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 4733
I20260929 16:00:34.679155 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 4335
I20260929 16:00:34.680517 4398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:34.786988 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 4335
I20260929 16:00:34.808605 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37461
--webserver_interface=127.23.222.126
--webserver_port=37941
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:37461
--ranger_config_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:35125/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.222.84:35125/kms with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:35.133833 4874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:35.134209 4874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:35.134371 4874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:35.141551 4874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:35.141701 4874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:35.141809 4874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:35.141913 4874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:35.141999 4874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:35.152184 4874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.222.84:35125/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:37461
--ranger_config_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=37941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:35.153514 4874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:35.154892 4874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:35.166569 4882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:35.166662 4879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:35.166637 4880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:35.167359 4874 server_base.cc:1034] running on GCE node
I20260929 16:00:35.168164 4874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:35.169643 4874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:35.170876 4874 hybrid_clock.cc:648] HybridClock initialized: now 1790697635170838 us; error 47 us; skew 500 ppm
I20260929 16:00:35.171294 4874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:35 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:35.174363 4874 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:00:35.175863 4874 webserver.cc:460] Webserver started at http://127.23.222.126:37941/ using document root <none> and password file <none>
I20260929 16:00:35.176523 4874 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:35.176673 4874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:00:35 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:35.247686 4874 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 16:00:35.250926 4890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:35.252446 4874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:00:35.252653 4874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd"
format_stamp: "Formatted at 2026-09-29 16:00:31 on dist-test-slave-cq88"
server_key: "06e3c55892c6c1674225712d3c78f3ff"
server_key_iv: "cf116886b645e75021a9c80f6062d165"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:35.253679 4874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:35.281277 4874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:35.285581 4874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:35.285910 4874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:35.304689 4874 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:37461
I20260929 16:00:35.304708 4941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:37461 every 8 connection(s)
I20260929 16:00:35.306255 4874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:00:35.311729 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 4874
I20260929 16:00:35.312775 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:35125/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:45037
--local_ip_for_outbound_sockets=127.23.222.65
--tserver_master_addrs=127.23.222.126:37461
--webserver_port=41939
--webserver_interface=127.23.222.65
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.222.84:35125/kms with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:35.313905 4942 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:00:35.319327 4942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Bootstrap starting.
I20260929 16:00:35.339574 4942 log.cc:824] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:35.348644 4942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:00:35.349200 4942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Bootstrap complete.
I20260929 16:00:35.358510 4942 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } }
I20260929 16:00:35.359524 4942 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2e7f58ee1ea4a20b46f54bbad4c83dd, State: Initialized, Role: FOLLOWER
I20260929 16:00:35.360193 4942 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } }
I20260929 16:00:35.360523 4942 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:00:35.360657 4942 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:00:35.360841 4942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:00:35.364759 4942 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } }
I20260929 16:00:35.365262 4942 leader_election.cc:302] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2e7f58ee1ea4a20b46f54bbad4c83dd; no voters:
I20260929 16:00:35.365983 4942 leader_election.cc:288] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:00:35.366148 4946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:00:35.367349 4946 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [term 2 LEADER]: Becoming Leader. State: Replica: d2e7f58ee1ea4a20b46f54bbad4c83dd, State: Running, Role: LEADER
I20260929 16:00:35.367975 4946 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } }
I20260929 16:00:35.368304 4942 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:35.370991 4948 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } } }
I20260929 16:00:35.371385 4948 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:35.371032 4949 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2e7f58ee1ea4a20b46f54bbad4c83dd. Latest consensus state: current_term: 2 leader_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2e7f58ee1ea4a20b46f54bbad4c83dd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37461 } } }
I20260929 16:00:35.371496 4949 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:35.384372 4953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:00:35.391757 4953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:00:35.393790 4953 catalog_manager.cc:1259] Loaded cluster ID: 634cdde9f20941bbaeb7d9532726d1c3
I20260929 16:00:35.394032 4953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:00:35.399863 4953 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:00:35.402292 4953 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Loaded TSK: 0
I20260929 16:00:35.404503 4953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:00:35.409602 4942 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 16:00:35.774961 4944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:35.775445 4944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:35.775642 4944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:35.787717 4944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:35.787951 4944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:35.788182 4944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:00:35.804379 4944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.222.84:35125/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:45037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=41939
--tserver_master_addrs=127.23.222.126:37461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:35.806165 4944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:35.807999 4944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:35.843310 4980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:35.843820 4981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:35.849059 4944 server_base.cc:1034] running on GCE node
W20260929 16:00:35.850067 4983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:35.850968 4944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:35.858531 4944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:35.865643 4944 hybrid_clock.cc:648] HybridClock initialized: now 1790697635865499 us; error 133 us; skew 500 ppm
I20260929 16:00:35.866266 4944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:35 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:35.872675 4944 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:00:35.881848 4944 webserver.cc:460] Webserver started at http://127.23.222.65:41939/ using document root <none> and password file <none>
I20260929 16:00:35.882982 4944 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:35.883169 4944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:00:35 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:35.954217 4944 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.003s sys 0.012s
I20260929 16:00:35.967432 4992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:35.970188 4944 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.007s sys 0.005s
I20260929 16:00:35.970510 4944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "757a837d0cef49a5a1e125aff38b118b"
format_stamp: "Formatted at 2026-09-29 16:00:32 on dist-test-slave-cq88"
server_key: "0983d691e27055058b80c6ef9bc50fc8"
server_key_iv: "ee2c20a92cbf988fcd2e1929d983f169"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:35.972136 4944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:36.006582 4944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:36.012383 4944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:36.012728 4944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:36.015388 4944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:36.017117 4944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:36.017235 4944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:36.017383 4944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:36.017472 4944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:36.069664 4944 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:45037
I20260929 16:00:36.069756 5104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:45037 every 8 connection(s)
I20260929 16:00:36.071568 4944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:00:36.072583 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 4944
I20260929 16:00:36.073376 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:35125/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:46287
--local_ip_for_outbound_sockets=127.23.222.66
--tserver_master_addrs=127.23.222.126:37461
--webserver_port=40639
--webserver_interface=127.23.222.66
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.222.84:35125/kms with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:36 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697635, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:36.125213 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:36.081372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37461 (local address 127.23.222.65:59379)
0929 16:00:36.084732 (+ 3360us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:36.084798 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:36.088192 (+ 3394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:36.105393 (+ 17201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:36.105406 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:36.105982 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:36.106699 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:36.106720 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:36.108803 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:36.108807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:36.110329 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:36.110347 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:36.111537 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:36.111545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:36.111837 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:36.112582 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:36.112606 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:36.114967 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:36.119816 (+ 4849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:36.119827 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:36.119844 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:36.120463 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:36.121099 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:36.121104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:36.121108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:36.121378 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:36.122452 (+ 1074us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:36.122460 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:36.122868 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 16:00:36.123229 (+ 361us) client_negotiation.cc:240] Negotiation successful
0929 16:00:36.123833 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2949,"thread_start_us":2620,"threads_started":1}
I20260929 16:00:36.127701 5109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:36.081758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:59379 (local address 127.23.222.126:37461)
0929 16:00:36.103430 (+ 21672us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:36.103470 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:36.103544 (+ 74us) server_negotiation.cc:407] Connection header received
0929 16:00:36.104088 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:36.104107 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:36.104625 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:36.105183 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:36.106933 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:36.108556 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:36.110520 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:36.111228 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:36.111519 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:36.115351 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:36.115363 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:36.115384 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:36.115433 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:36.119607 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:36.120700 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:36.120710 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:36.120717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:36.120938 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:36.121565 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:36.121570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:36.121573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:36.122275 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:36.122508 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:36.125900 (+ 3392us) server_negotiation.cc:299] Negotiation successful
0929 16:00:36.126367 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17957,"thread_start_us":17693,"threads_started":1}
I20260929 16:00:36.130936 5105 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37461
I20260929 16:00:36.131296 5105 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:36.132184 5105 heartbeater.cc:511] Master 127.23.222.126:37461 requested a full tablet report, sending...
W20260929 16:00:36.135478 5105 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:37461 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:00:36.135715 5105 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:36.136013 5105 heartbeater.cc:511] Master 127.23.222.126:37461 requested a full tablet report, sending...
I20260929 16:00:36.137357 5105 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:36.137650 5105 heartbeater.cc:511] Master 127.23.222.126:37461 requested a full tablet report, sending...
W20260929 16:00:36.138710 5105 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:00:36.518523 5107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:36.518857 5107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:36.519027 5107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:36.526746 5107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:36.526893 5107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:36.527067 5107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:00:36.538693 5107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.222.84:35125/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:46287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=40639
--tserver_master_addrs=127.23.222.126:37461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:36.540673 5107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:36.542464 5107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:36.589119 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:36.589968 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:36.599442 5128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:36.600687 5107 server_base.cc:1034] running on GCE node
I20260929 16:00:36.601733 5107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:36.603120 5107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:36.604488 5107 hybrid_clock.cc:648] HybridClock initialized: now 1790697636604405 us; error 68 us; skew 500 ppm
I20260929 16:00:36.604887 5107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:36 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:36.609293 5107 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:00:36.612946 5107 webserver.cc:460] Webserver started at http://127.23.222.66:40639/ using document root <none> and password file <none>
I20260929 16:00:36.614046 5107 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:36.614260 5107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:00:36 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:36.648034 5107 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.001s
I20260929 16:00:36.667969 5136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:36.670166 5107 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.008s sys 0.011s
Sep 29 16:00:36 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790697636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:36.670463 5107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "2d137b12f8a24d9382203509f0f05b2b"
format_stamp: "Formatted at 2026-09-29 16:00:33 on dist-test-slave-cq88"
server_key: "6342c777ef3df81ac66ce35004879aa9"
server_key_iv: "e5b088f384e11dadbe35074395a5724f"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:36.672353 5107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:36.744171 5107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:36.749441 5107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:36.749845 5107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:36.752115 5107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:36.753798 5107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:36.753929 5107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:36.754071 5107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:36.754176 5107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:36.814785 5107 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:46287
I20260929 16:00:36.814838 5252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:46287 every 8 connection(s)
I20260929 16:00:36.816490 5107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:00:36.822917 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 5107
I20260929 16:00:36.823770 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.222.84:35125/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:36543
--local_ip_for_outbound_sockets=127.23.222.67
--tserver_master_addrs=127.23.222.126:37461
--webserver_port=33427
--webserver_interface=127.23.222.67
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.222.84:35125/kms with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:36 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:36.883493 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:36.820938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37461 (local address 127.23.222.66:52919)
0929 16:00:36.842158 (+ 21220us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:36.842226 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:36.846439 (+ 4213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:36.847827 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:36.847836 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:36.848335 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:36.848944 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:36.848963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:36.850110 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:36.850116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:36.851513 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:36.851528 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:36.852378 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:36.852387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:36.852642 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:36.860102 (+ 7460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:36.860154 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:36.863203 (+ 3049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:36.871564 (+ 8361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:36.871576 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:36.871600 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:36.873336 (+ 1736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:36.873927 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:36.873932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:36.873936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:36.874285 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:36.880389 (+ 6104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:36.880407 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:36.880974 (+ 567us) client_negotiation.cc:769] Sending connection context
0929 16:00:36.881327 (+ 353us) client_negotiation.cc:240] Negotiation successful
0929 16:00:36.881976 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20799,"thread_start_us":20519,"threads_started":1}
I20260929 16:00:36.885174 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:36.821734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:52919 (local address 127.23.222.126:37461)
0929 16:00:36.841414 (+ 19680us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:36.841421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:36.842903 (+ 1482us) server_negotiation.cc:407] Connection header received
0929 16:00:36.846979 (+ 4076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:36.846989 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:36.847134 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:36.847223 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:36.849161 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:36.849923 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:36.851680 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:36.852109 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:36.852258 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:36.863615 (+ 11357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:36.863624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:36.863629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:36.863683 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:36.871294 (+ 7611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:36.873557 (+ 2263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:36.873564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:36.873568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:36.873761 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:36.874442 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:36.874447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:36.874451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:36.876978 (+ 2527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:36.884269 (+ 7291us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:36.884529 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 16:00:36.884811 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19504,"thread_start_us":19367,"threads_started":1}
I20260929 16:00:36.887418 5253 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37461
I20260929 16:00:36.887763 5253 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:36.888633 5253 heartbeater.cc:511] Master 127.23.222.126:37461 requested a full tablet report, sending...
I20260929 16:00:36.890858 4907 ts_manager.cc:194] Registered new tserver with Master: 2d137b12f8a24d9382203509f0f05b2b (127.23.222.66:46287)
I20260929 16:00:36.893033 4907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:52919
I20260929 16:00:37.138867 5105 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:37.139467 5105 heartbeater.cc:511] Master 127.23.222.126:37461 requested a full tablet report, sending...
I20260929 16:00:37.140720 4907 ts_manager.cc:194] Registered new tserver with Master: 757a837d0cef49a5a1e125aff38b118b (127.23.222.65:45037)
I20260929 16:00:37.143121 4907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:59379
Sep 29 16:00:37 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
W20260929 16:00:37.311034 5258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:37.311555 5258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:37.311746 5258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:37.323374 5258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:37.323550 5258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:37.323773 5258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:00:37.339686 5258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.222.84:35125/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:36543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=33427
--tserver_master_addrs=127.23.222.126:37461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:37.341710 5258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:37.343562 5258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:37.388337 5263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:37.395627 5264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:37.396028 5258 server_base.cc:1034] running on GCE node
W20260929 16:00:37.396704 5267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:37.397559 5258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:37.404078 5258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:37.405364 5258 hybrid_clock.cc:648] HybridClock initialized: now 1790697637405291 us; error 72 us; skew 500 ppm
I20260929 16:00:37.405864 5258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:37 dist-test-slave-cq88 krb5kdc[3888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:37.414570 5258 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:00:37.459867 5258 webserver.cc:460] Webserver started at http://127.23.222.67:33427/ using document root <none> and password file <none>
I20260929 16:00:37.461061 5258 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:37.461320 5258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 16:00:37 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:37.516516 5258 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.002s sys 0.003s
I20260929 16:00:37.531579 5274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:37.533905 5258 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.010s sys 0.006s
I20260929 16:00:37.534235 5258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "7fbae2062b09427b89f2235dde034d55"
format_stamp: "Formatted at 2026-09-29 16:00:34 on dist-test-slave-cq88"
server_key: "505663573bb123012664265e749f5b51"
server_key_iv: "9268ea23c395a6c89c5bee10a8cf18b0"
server_key_version: "kuduclusterkey@0"
I20260929 16:00:37.535815 5258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 16:00:37 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697636, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for HTTP/127.23.222.84@KRBTEST.COM
I20260929 16:00:37.589313 5258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:37.594750 5258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:37.595142 5258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:37.611434 5258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:37.620528 5258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:37.620754 5258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 16:00:37.620909 5258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:37.621026 5258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:37.855758 5258 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:36543
I20260929 16:00:37.855803 5390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:36543 every 8 connection(s)
I20260929 16:00:37.857519 5258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:00:37.867515 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 5258
Sep 29 16:00:37 dist-test-slave-cq88 krb5kdc[3888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697637, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:37.891654 5394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.861139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:49365 (local address 127.23.222.126:37461)
0929 16:00:37.868839 (+ 7700us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:37.868848 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:37.869886 (+ 1038us) server_negotiation.cc:407] Connection header received
0929 16:00:37.873360 (+ 3474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:37.873367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:37.873505 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:37.873644 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:37.876102 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.877210 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.879755 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.880164 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.880300 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:37.884230 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:37.884235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:37.884238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:37.884274 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:37.887883 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:37.888898 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:37.888904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:37.888907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:37.889060 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:37.889623 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:37.889628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:37.889631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:37.889905 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:37.890063 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:37.890943 (+ 880us) server_negotiation.cc:299] Negotiation successful
0929 16:00:37.891215 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7543,"thread_start_us":13600,"threads_started":1}
I20260929 16:00:37.892746 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.861661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37461 (local address 127.23.222.67:49365)
0929 16:00:37.869532 (+ 7871us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:37.869745 (+ 213us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:37.872930 (+ 3185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:37.874176 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:37.874190 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:37.874894 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:37.875887 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:37.875916 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.877433 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.877437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:37.879610 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:37.879621 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.880411 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.880419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:37.880654 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:37.881387 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:37.881429 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:37.883994 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:37.888072 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:37.888085 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:37.888107 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:37.888753 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:37.889220 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:37.889226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:37.889230 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:37.889504 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:37.890052 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:37.890060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:37.890507 (+ 447us) client_negotiation.cc:769] Sending connection context
0929 16:00:37.890812 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 16:00:37.891415 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7476,"thread_start_us":7258,"threads_started":1}
I20260929 16:00:37.893151 5398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.869111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.23.222.126:37461)
0929 16:00:37.875436 (+ 6325us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:37.875442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:37.875462 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:00:37.875515 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:37.875520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:37.875611 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:37.875715 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:37.877993 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.879322 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.881423 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.882087 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.882211 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:37.885627 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:37.885639 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:37.885645 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:37.885715 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:37.889285 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:37.890109 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:37.890113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:37.890116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:37.890240 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:37.890688 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:37.890693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:37.890696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:37.891004 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:37.891292 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:37.892556 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0929 16:00:37.892782 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6202,"thread_start_us":6031,"threads_started":1}
I20260929 16:00:37.894737 5391 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37461
I20260929 16:00:37.895062 5391 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:37.895859 5391 heartbeater.cc:511] Master 127.23.222.126:37461 requested a full tablet report, sending...
I20260929 16:00:37.897444 4907 ts_manager.cc:194] Registered new tserver with Master: 7fbae2062b09427b89f2235dde034d55 (127.23.222.67:36543)
I20260929 16:00:37.897745 5253 heartbeater.cc:503] Master 127.23.222.126:37461 was elected leader, sending a full tablet report...
I20260929 16:00:37.898540 4907 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:49365
I20260929 16:00:37.913125 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:00:37.932494 5398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.917028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.23.222.126:37461)
0929 16:00:37.917246 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:37.917252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:37.917727 (+ 475us) server_negotiation.cc:407] Connection header received
0929 16:00:37.917877 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:37.917882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:37.917962 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:37.918071 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:37.919823 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.920931 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.922406 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.923070 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.923212 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:37.925943 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:37.925948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:37.925953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:37.925996 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:37.929425 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:37.930150 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:37.930156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:37.930160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:37.930326 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:37.930788 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:37.930793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:37.930797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:37.931157 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:37.931425 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:37.931903 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 16:00:37.932152 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:00:37.938400 4906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:00:37.949079 4906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:00:37.995669 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.965135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45037 (local address 127.0.0.1:45746)
0929 16:00:37.966361 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:37.966408 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:37.966608 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:37.974564 (+ 7956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:37.974583 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:37.974633 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:37.975211 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:37.975230 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.986263 (+ 11033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.986274 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:37.990987 (+ 4713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:37.991013 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.994659 (+ 3646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.994666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:37.994873 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:37.994925 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 16:00:37.995049 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 16:00:37.995306 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"thread_start_us":27681,"threads_started":1}
I20260929 16:00:37.999313 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.965664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45746 (local address 127.23.222.65:45037)
0929 16:00:37.967307 (+ 1643us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:37.967319 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:37.967357 (+ 38us) server_negotiation.cc:407] Connection header received
0929 16:00:37.967480 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:37.967492 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:37.969133 (+ 1641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:37.979582 (+ 10449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:37.979970 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.985111 (+ 5141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.991319 (+ 6208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.994394 (+ 3075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.994568 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:37.998564 (+ 3996us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:37.998676 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 16:00:37.998858 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1403,"thread_start_us":1257,"threads_started":1}
I20260929 16:00:38.001828 5411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.965537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:36543 (local address 127.0.0.1:46974)
0929 16:00:37.993186 (+ 27649us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:37.993219 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:37.993388 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:37.994095 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:37.994100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:37.994130 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:37.994474 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:37.994490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.996509 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.996514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:37.998812 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:37.998826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.001013 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.001019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:38.001204 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.001240 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 16:00:38.001338 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:00:38.001466 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27456,"thread_start_us":942,"threads_started":1}
I20260929 16:00:38.002635 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.966207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.23.222.67:36543)
0929 16:00:37.991647 (+ 25440us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:37.991656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:37.993369 (+ 1713us) server_negotiation.cc:407] Connection header received
0929 16:00:37.993548 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:37.993557 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:37.993847 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:37.994069 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:37.994717 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.996356 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.998997 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.000493 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.000620 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.000866 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:38.002147 (+ 1281us) server_negotiation.cc:299] Negotiation successful
0929 16:00:38.002306 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25005,"thread_start_us":25035,"threads_started":1}
I20260929 16:00:38.003063 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.969596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:46287 (local address 127.0.0.1:53732)
0929 16:00:37.996171 (+ 26575us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:37.996191 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:37.996289 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:37.996947 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:37.996952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:37.996977 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:37.997183 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:37.997192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.998978 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:37.998983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:38.000593 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:38.000607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.002368 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.002374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:38.002552 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.002586 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:00:38.002646 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 16:00:38.002777 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26470,"thread_start_us":28633,"threads_started":1}
I20260929 16:00:38.004151 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:37.969820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.23.222.66:46287)
0929 16:00:37.992791 (+ 22971us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:37.992802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:37.996271 (+ 3469us) server_negotiation.cc:407] Connection header received
0929 16:00:37.996421 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:37.996433 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:37.996737 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:37.996944 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:37.997372 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:37.998830 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.000816 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.002181 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.003328 (+ 1147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.003557 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:38.003628 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 16:00:38.003806 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22785,"thread_start_us":22579,"threads_started":1}
I20260929 16:00:38.003930 5040 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9cd06eb7314b439ebfc2c021e0847a (DEFAULT_TABLE table=test-table [id=1e68d1884edf42d5a4ed1ef0eb067659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:38.005203 5040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9cd06eb7314b439ebfc2c021e0847a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:38.005743 5323 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9cd06eb7314b439ebfc2c021e0847a (DEFAULT_TABLE table=test-table [id=1e68d1884edf42d5a4ed1ef0eb067659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:38.006965 5323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9cd06eb7314b439ebfc2c021e0847a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:38.010270 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9cd06eb7314b439ebfc2c021e0847a (DEFAULT_TABLE table=test-table [id=1e68d1884edf42d5a4ed1ef0eb067659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:38.011693 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9cd06eb7314b439ebfc2c021e0847a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:38.025751 5418 tablet_bootstrap.cc:492] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: Bootstrap starting.
I20260929 16:00:38.031392 5417 tablet_bootstrap.cc:492] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: Bootstrap starting.
I20260929 16:00:38.032645 5419 tablet_bootstrap.cc:492] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: Bootstrap starting.
I20260929 16:00:38.035120 5417 tablet_bootstrap.cc:654] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:38.036203 5418 tablet_bootstrap.cc:654] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:38.036198 5419 tablet_bootstrap.cc:654] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:38.036507 5417 log.cc:824] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:38.037456 5419 log.cc:824] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:38.037781 5418 log.cc:824] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:38.045449 5419 tablet_bootstrap.cc:492] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: No bootstrap required, opened a new log
I20260929 16:00:38.045945 5419 ts_tablet_manager.cc:1397] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260929 16:00:38.058749 5418 tablet_bootstrap.cc:492] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: No bootstrap required, opened a new log
I20260929 16:00:38.058828 5417 tablet_bootstrap.cc:492] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: No bootstrap required, opened a new log
I20260929 16:00:38.059175 5418 ts_tablet_manager.cc:1397] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.000s
I20260929 16:00:38.059170 5417 ts_tablet_manager.cc:1397] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.021s
I20260929 16:00:38.061698 5419 raft_consensus.cc:348] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } }
I20260929 16:00:38.062340 5419 raft_consensus.cc:374] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:38.062484 5419 raft_consensus.cc:729] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d137b12f8a24d9382203509f0f05b2b, State: Initialized, Role: FOLLOWER
I20260929 16:00:38.063405 5419 consensus_queue.cc:260] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } }
I20260929 16:00:38.069555 5418 raft_consensus.cc:348] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } }
I20260929 16:00:38.070147 5418 raft_consensus.cc:374] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:38.070294 5418 raft_consensus.cc:729] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757a837d0cef49a5a1e125aff38b118b, State: Initialized, Role: FOLLOWER
I20260929 16:00:38.071121 5418 consensus_queue.cc:260] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } }
I20260929 16:00:38.071583 5419 ts_tablet_manager.cc:1428] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: Time spent starting tablet: real 0.025s user 0.018s sys 0.004s
I20260929 16:00:38.080267 5105 heartbeater.cc:503] Master 127.23.222.126:37461 was elected leader, sending a full tablet report...
I20260929 16:00:38.080987 5418 ts_tablet_manager.cc:1428] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
W20260929 16:00:38.083945 5106 tablet.cc:2369] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:00:38.083951 5417 raft_consensus.cc:348] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } }
I20260929 16:00:38.084399 5417 raft_consensus.cc:374] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:38.084491 5417 raft_consensus.cc:729] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fbae2062b09427b89f2235dde034d55, State: Initialized, Role: FOLLOWER
I20260929 16:00:38.085122 5417 consensus_queue.cc:260] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } }
I20260929 16:00:38.086187 5391 heartbeater.cc:503] Master 127.23.222.126:37461 was elected leader, sending a full tablet report...
I20260929 16:00:38.086863 5417 ts_tablet_manager.cc:1428] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: Time spent starting tablet: real 0.028s user 0.006s sys 0.007s
W20260929 16:00:38.098718 5254 tablet.cc:2369] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:00:38.111294 5392 tablet.cc:2369] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:00:38.284162 5425 raft_consensus.cc:482] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:00:38.284449 5425 raft_consensus.cc:504] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } }
I20260929 16:00:38.286105 5425 leader_election.cc:288] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 757a837d0cef49a5a1e125aff38b118b (127.23.222.65:45037), 2d137b12f8a24d9382203509f0f05b2b (127.23.222.66:46287)
I20260929 16:00:38.293007 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:38.286860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:44449 (local address 127.23.222.66:46287)
0929 16:00:38.287086 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:38.287094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:38.287116 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:00:38.287201 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:38.287206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:38.287329 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:38.287435 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:38.288038 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.289072 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.290924 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.292114 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.292237 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.292292 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:38.292652 (+ 360us) server_negotiation.cc:299] Negotiation successful
0929 16:00:38.292803 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:00:38.293085 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:38.286409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:46287 (local address 127.23.222.67:44449)
0929 16:00:38.286621 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:38.286649 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:38.286786 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:38.287494 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:38.287498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:38.287525 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:38.287801 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:38.287814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.289219 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.289224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:38.290757 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:38.290769 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.292276 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.292282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:38.292469 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.292508 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 16:00:38.292627 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 16:00:38.292793 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"spinlock_wait_cycles":32000}
I20260929 16:00:38.294130 5426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:38.286528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45037 (local address 127.23.222.67:38903)
0929 16:00:38.287498 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:38.287515 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:38.287645 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:38.288047 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:38.288053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:38.288086 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:38.288545 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:38.288559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.289920 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.289924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:38.291708 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:38.291727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.293420 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.293427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:38.293595 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.293625 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:00:38.293724 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:00:38.293860 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":4,"thread_start_us":554,"threads_started":1}
I20260929 16:00:38.294201 5208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9cd06eb7314b439ebfc2c021e0847a" candidate_uuid: "7fbae2062b09427b89f2235dde034d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d137b12f8a24d9382203509f0f05b2b" is_pre_election: true
I20260929 16:00:38.294850 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:38.286684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:38903 (local address 127.23.222.65:45037)
0929 16:00:38.286924 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:38.286931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:38.287617 (+ 686us) server_negotiation.cc:407] Connection header received
0929 16:00:38.287777 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:38.287783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:38.287874 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:38.287971 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:38.288723 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.289718 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.291895 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.293041 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.293233 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.293314 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:38.294581 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0929 16:00:38.294686 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:00:38.294822 5208 raft_consensus.cc:2454] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fbae2062b09427b89f2235dde034d55 in term 0.
I20260929 16:00:38.295536 5277 leader_election.cc:302] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d137b12f8a24d9382203509f0f05b2b, 7fbae2062b09427b89f2235dde034d55; no voters:
I20260929 16:00:38.295807 5060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9cd06eb7314b439ebfc2c021e0847a" candidate_uuid: "7fbae2062b09427b89f2235dde034d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757a837d0cef49a5a1e125aff38b118b" is_pre_election: true
I20260929 16:00:38.295956 5425 raft_consensus.cc:2781] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:00:38.296099 5425 raft_consensus.cc:482] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:00:38.296224 5425 raft_consensus.cc:3037] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:38.296351 5060 raft_consensus.cc:2454] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fbae2062b09427b89f2235dde034d55 in term 0.
I20260929 16:00:38.298744 5425 raft_consensus.cc:504] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } }
I20260929 16:00:38.299503 5425 leader_election.cc:288] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [CANDIDATE]: Term 1 election: Requested vote from peers 757a837d0cef49a5a1e125aff38b118b (127.23.222.65:45037), 2d137b12f8a24d9382203509f0f05b2b (127.23.222.66:46287)
I20260929 16:00:38.299938 5060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9cd06eb7314b439ebfc2c021e0847a" candidate_uuid: "7fbae2062b09427b89f2235dde034d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757a837d0cef49a5a1e125aff38b118b"
I20260929 16:00:38.300110 5208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9cd06eb7314b439ebfc2c021e0847a" candidate_uuid: "7fbae2062b09427b89f2235dde034d55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d137b12f8a24d9382203509f0f05b2b"
I20260929 16:00:38.300176 5060 raft_consensus.cc:3037] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:38.300336 5208 raft_consensus.cc:3037] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:38.303511 5060 raft_consensus.cc:2454] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fbae2062b09427b89f2235dde034d55 in term 1.
I20260929 16:00:38.303514 5208 raft_consensus.cc:2454] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fbae2062b09427b89f2235dde034d55 in term 1.
I20260929 16:00:38.304033 5278 leader_election.cc:302] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d137b12f8a24d9382203509f0f05b2b, 757a837d0cef49a5a1e125aff38b118b, 7fbae2062b09427b89f2235dde034d55; no voters:
I20260929 16:00:38.304373 5425 raft_consensus.cc:2781] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:00:38.304941 5425 raft_consensus.cc:686] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 1 LEADER]: Becoming Leader. State: Replica: 7fbae2062b09427b89f2235dde034d55, State: Running, Role: LEADER
I20260929 16:00:38.305709 5425 consensus_queue.cc:237] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } }
I20260929 16:00:38.311101 4906 catalog_manager.cc:5495] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fbae2062b09427b89f2235dde034d55 (127.23.222.67). New cstate: current_term: 1 leader_uuid: "7fbae2062b09427b89f2235dde034d55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7fbae2062b09427b89f2235dde034d55" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 36543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 } health_report { overall_health: UNKNOWN } } }
I20260929 16:00:38.353483 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:38.347732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.23.222.67:36543)
0929 16:00:38.347954 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:38.347960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:38.348069 (+ 109us) server_negotiation.cc:407] Connection header received
0929 16:00:38.348363 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:38.348367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:38.348459 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:38.348667 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:00:38.349046 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.349883 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.351313 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.351710 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.351847 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.352354 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:00:38.352874 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:00:38.353038 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:38.353132 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 16:00:38.353213 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:00:38.362322 5208 raft_consensus.cc:1261] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [term 1 FOLLOWER]: Refusing update from remote peer 7fbae2062b09427b89f2235dde034d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:38.362321 5060 raft_consensus.cc:1261] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [term 1 FOLLOWER]: Refusing update from remote peer 7fbae2062b09427b89f2235dde034d55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:38.363592 5425 consensus_queue.cc:1035] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d137b12f8a24d9382203509f0f05b2b" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 46287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:00:38.363941 5427 consensus_queue.cc:1035] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757a837d0cef49a5a1e125aff38b118b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:00:38.381474 5431 mvcc.cc:205] Tried to move back new op lower bound from 7334697526720221184 to 7334697526504448000. Current Snapshot: MvccSnapshot[applied={T|T < 7334697526720221184}]
I20260929 16:00:38.392508 4906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:00:38.392807 4906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:00:38.397010 4906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2e7f58ee1ea4a20b46f54bbad4c83dd: Sending DeleteTablet for 3 replicas of tablet 8b9cd06eb7314b439ebfc2c021e0847a
I20260929 16:00:38.398285 5323 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9cd06eb7314b439ebfc2c021e0847a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:38 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:46974
I20260929 16:00:38.398315 5040 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9cd06eb7314b439ebfc2c021e0847a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:38 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:45746
I20260929 16:00:38.398396 5188 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9cd06eb7314b439ebfc2c021e0847a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:38 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:53732
I20260929 16:00:38.399283 5440 tablet_replica.cc:330] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: stopping tablet replica
I20260929 16:00:38.399749 5440 raft_consensus.cc:2229] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:00:38.400439 5440 raft_consensus.cc:2258] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:38.401101 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 4944
I20260929 16:00:38.403504 5440 ts_tablet_manager.cc:1905] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:38.406771 5441 tablet_replica.cc:330] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: stopping tablet replica
I20260929 16:00:38.407392 5441 raft_consensus.cc:2229] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:00:38.407989 5441 raft_consensus.cc:2258] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:38.408675 5442 tablet_replica.cc:330] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: stopping tablet replica
I20260929 16:00:38.410109 5442 raft_consensus.cc:2229] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:00:38.410678 5442 raft_consensus.cc:2258] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:38.411296 5440 ts_tablet_manager.cc:1918] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:38.411602 5440 log.cc:1196] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/8b9cd06eb7314b439ebfc2c021e0847a
I20260929 16:00:38.412012 5441 ts_tablet_manager.cc:1905] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:38.412221 5440 ts_tablet_manager.cc:1939] T 8b9cd06eb7314b439ebfc2c021e0847a P 7fbae2062b09427b89f2235dde034d55: Deleting consensus metadata
I20260929 16:00:38.413259 5442 ts_tablet_manager.cc:1905] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:38.414060 4891 catalog_manager.cc:4841] TS 7fbae2062b09427b89f2235dde034d55 (127.23.222.67:36543): tablet 8b9cd06eb7314b439ebfc2c021e0847a (table test-table [id=1e68d1884edf42d5a4ed1ef0eb067659]) successfully deleted
I20260929 16:00:38.419559 5441 ts_tablet_manager.cc:1918] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:38.419829 5441 log.cc:1196] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/8b9cd06eb7314b439ebfc2c021e0847a
I20260929 16:00:38.420463 5441 ts_tablet_manager.cc:1939] T 8b9cd06eb7314b439ebfc2c021e0847a P 2d137b12f8a24d9382203509f0f05b2b: Deleting consensus metadata
I20260929 16:00:38.421922 5442 ts_tablet_manager.cc:1918] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:38.422149 5442 log.cc:1196] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/8b9cd06eb7314b439ebfc2c021e0847a
I20260929 16:00:38.422612 4892 catalog_manager.cc:4841] TS 2d137b12f8a24d9382203509f0f05b2b (127.23.222.66:46287): tablet 8b9cd06eb7314b439ebfc2c021e0847a (table test-table [id=1e68d1884edf42d5a4ed1ef0eb067659]) successfully deleted
I20260929 16:00:38.422807 5442 ts_tablet_manager.cc:1939] T 8b9cd06eb7314b439ebfc2c021e0847a P 757a837d0cef49a5a1e125aff38b118b: Deleting consensus metadata
I20260929 16:00:38.424445 4893 catalog_manager.cc:4841] TS 757a837d0cef49a5a1e125aff38b118b (127.23.222.65:45037): tablet 8b9cd06eb7314b439ebfc2c021e0847a (table test-table [id=1e68d1884edf42d5a4ed1ef0eb067659]) successfully deleted
I20260929 16:00:38.427558 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:38.402078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.23.222.65:45037)
0929 16:00:38.402300 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:38.402309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:38.409280 (+ 6971us) server_negotiation.cc:407] Connection header received
0929 16:00:38.409506 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:38.409513 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:38.409597 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:38.409707 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:38.411995 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.413119 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.415202 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.415893 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.416006 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.419730 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:38.419751 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:38.419770 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:38.419825 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:38.424261 (+ 4436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:38.425239 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:38.425246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:38.425249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:38.425379 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:38.425813 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:38.425817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:38.425819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:38.426193 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:38.426426 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:38.426977 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 16:00:38.427163 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:00:38.428332 5100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:38.521042 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 4944
I20260929 16:00:38.538621 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 5107
I20260929 16:00:38.552976 5416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:38.539455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.23.222.66:46287)
0929 16:00:38.539678 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:38.539685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:38.539708 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:00:38.539826 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:38.539832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:38.539925 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:38.540040 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:38.541327 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.542469 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.543969 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.544401 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.544544 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.547138 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:38.547153 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:38.547170 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:38.547207 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:38.550152 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:38.550827 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:38.550836 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:38.550840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:38.550975 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:38.551434 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:38.551441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:38.551444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:38.551784 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:38.552119 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:38.552388 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:00:38.552613 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:00:38.553606 5248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:38.679324 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 5107
I20260929 16:00:38.696389 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 5258
I20260929 16:00:38.713821 5414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:38.697312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.23.222.67:36543)
0929 16:00:38.697496 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:38.697500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:38.697567 (+ 67us) server_negotiation.cc:407] Connection header received
0929 16:00:38.697768 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:38.697772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:38.697864 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:38.697948 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:38.700028 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.700931 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.702898 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:38.703475 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:38.703618 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:38.707173 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:38.707186 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:38.707202 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:38.707295 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:38.710475 (+ 3180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:38.711612 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:38.711620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:38.711622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:38.711777 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:38.712225 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:38.712228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:38.712230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:38.712630 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:38.712874 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:38.713305 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 16:00:38.713475 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:00:38.714696 5386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:38.837575 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 5258
I20260929 16:00:38.857591 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 4874
I20260929 16:00:38.858958 4937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:38.974217 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 4874
I20260929 16:00:38.999709 24441 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 16:00:39.335521 24441 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 16:00:39.335731 24441 mini_ranger.cc:66] Stopping Ranger...
I20260929 16:00:39.681843 24441 mini_ranger.cc:68] Stopped Ranger
2026-09-29 16:00:39.682 UTC [3919] LOG: received smart shutdown request
2026-09-29 16:00:39.687 UTC [3919] LOG: background worker "logical replication launcher" (PID 3928) exited with exit code 1
2026-09-29 16:00:39.687 UTC [3923] LOG: shutting down
2026-09-29 16:00:39.889 UTC [3919] LOG: database system is shut down
2026-09-29T16:00:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55760 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:00:40 dist-test-slave-cq88 krb5kdc[5456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:00:40 dist-test-slave-cq88 krb5kdc[5456](info): set up 2 sockets
Sep 29 16:00:40 dist-test-slave-cq88 krb5kdc[5456](info): commencing operation
krb5kdc: starting...
W20260929 16:00:41.134446 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:00:41 dist-test-slave-cq88 krb5kdc[5456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:00:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:00:41.374951 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:36347
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:36347,127.23.222.125:45649,127.23.222.124:37125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:41.713122 5469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:41.713444 5469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:41.713538 5469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:41.721287 5469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:41.721422 5469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:41.721470 5469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:41.721529 5469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:41.721575 5469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:41.731935 5469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:36347,127.23.222.125:45649,127.23.222.124:37125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:36347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:41.733311 5469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:41.734688 5469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:41.746798 5476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:41.746833 5478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:41.747090 5475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:41.747660 5469 server_base.cc:1034] running on GCE node
I20260929 16:00:41.748485 5469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:41.750398 5469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:41.751638 5469 hybrid_clock.cc:648] HybridClock initialized: now 1790697641751591 us; error 48 us; skew 500 ppm
I20260929 16:00:41.752018 5469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:41 dist-test-slave-cq88 krb5kdc[5456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:41.755336 5469 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:00:41.756857 5469 webserver.cc:460] Webserver started at http://127.23.222.126:38791/ using document root <none> and password file <none>
I20260929 16:00:41.757542 5469 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:41.757649 5469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:41.757927 5469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:00:41.761199 5469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "001927311b244371a09c0a87e2849f48"
format_stamp: "Formatted at 2026-09-29 16:00:41 on dist-test-slave-cq88"
server_key: "dd754e2b7c65935347f6fc8e00d43039"
server_key_iv: "fa888f1a424102dd45222f1783353fca"
server_key_version: "encryptionkey@0"
I20260929 16:00:41.761987 5469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "001927311b244371a09c0a87e2849f48"
format_stamp: "Formatted at 2026-09-29 16:00:41 on dist-test-slave-cq88"
server_key: "dd754e2b7c65935347f6fc8e00d43039"
server_key_iv: "fa888f1a424102dd45222f1783353fca"
server_key_version: "encryptionkey@0"
I20260929 16:00:41.767657 5469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 16:00:41.771358 5485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:41.772773 5469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:00:41.772966 5469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "001927311b244371a09c0a87e2849f48"
format_stamp: "Formatted at 2026-09-29 16:00:41 on dist-test-slave-cq88"
server_key: "dd754e2b7c65935347f6fc8e00d43039"
server_key_iv: "fa888f1a424102dd45222f1783353fca"
server_key_version: "encryptionkey@0"
I20260929 16:00:41.773185 5469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:41.789336 5469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:41.793772 5469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:41.794063 5469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:41.812952 5469 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:36347
I20260929 16:00:41.812978 5536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:36347 every 8 connection(s)
I20260929 16:00:41.814484 5469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:00:41.818085 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 5469
I20260929 16:00:41.818323 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:00:41.819176 24441 external_mini_cluster.cc:1428] Setting key f75f6401564fb9796ddcd6a42afe1a13
I20260929 16:00:41.819393 5537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:41.829344 5537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.222.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.125@KRBTEST.COM" created.
Sep 29 16:00:41 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:41.852921 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:41.833003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:36347 (local address 127.0.0.1:48338)
0929 16:00:41.833951 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:41.834008 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:41.836149 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:41.837000 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:41.837007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:41.837737 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:41.838090 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:41.838108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:41.839682 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:41.839688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:41.840764 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:41.840775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:41.841754 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:41.841761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:41.841935 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:41.842625 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:41.842648 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:41.844972 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:41.848379 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:41.848393 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:41.848406 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:41.848826 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:41.849269 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:41.849276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:41.849280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:41.849439 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:41.850242 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:41.850250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:41.850475 (+ 225us) client_negotiation.cc:769] Sending connection context
0929 16:00:41.850739 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 16:00:41.851400 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":179,"thread_start_us":414,"threads_started":1}
I20260929 16:00:41.852943 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:41.833007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.23.222.126:36347)
0929 16:00:41.834830 (+ 1823us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:41.834839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:41.834888 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:00:41.836531 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:41.836537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:41.836781 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:41.836983 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:41.838279 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:41.839528 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:41.840909 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:41.841514 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:41.841724 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:41.845216 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:41.845225 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:41.845238 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:41.845270 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:41.848236 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:41.848980 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:41.848987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:41.848989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:41.849144 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:41.849551 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:41.849555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:41.849563 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:41.850094 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:41.850320 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:41.850845 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 16:00:41.851287 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":1604,"thread_start_us":1372,"threads_started":1}
I20260929 16:00:41.856434 5537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:00:41.857721 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:41.857309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59676)
0929 16:00:41.857476 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:41.857589 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 16:00:41.858003 5487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
I20260929 16:00:41.859810 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:41.859427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59680)
0929 16:00:41.859587 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:41.859658 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:00:41.860039 5537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:45649: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab.
Entry for principal kudu/127.23.222.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab.
I20260929 16:00:41.872925 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.125:45649
--webserver_interface=127.23.222.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab
--principal=kudu/127.23.222.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:36347,127.23.222.125:45649,127.23.222.124:37125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:41.909379 5537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } attempt: 1
I20260929 16:00:41.910681 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:41.910088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59682)
0929 16:00:41.910338 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:41.910518 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 16:00:41.911785 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:41.911390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59694)
0929 16:00:41.911537 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:41.911625 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 16:00:41.911974 5537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:45649: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
I20260929 16:00:41.980249 5537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } attempt: 2
I20260929 16:00:41.981662 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:41.981037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59706)
0929 16:00:41.981325 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:41.981462 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 16:00:41.982865 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:41.982394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59718)
0929 16:00:41.982582 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:41.982676 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 16:00:41.983119 5537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:45649: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
I20260929 16:00:42.074426 5537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } attempt: 3
I20260929 16:00:42.075769 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.075209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59730)
0929 16:00:42.075454 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.075586 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 16:00:42.076872 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.076463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59734)
0929 16:00:42.076613 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.076701 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:00:42.077133 5537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:45649: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
W20260929 16:00:42.211359 5544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:42.211737 5544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:42.211984 5544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:42.219357 5544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:42.219473 5544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:42.219535 5544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:42.219594 5544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:42.219637 5544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:42.220394 5537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } attempt: 4
I20260929 16:00:42.221649 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.221036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59748)
0929 16:00:42.221363 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.221479 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260929 16:00:42.222796 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.222358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:59752)
0929 16:00:42.222529 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.222641 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:00:42.223021 5537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:45649: Network error: Client connection negotiation failed: client connection to 127.23.222.125:45649: connect: Connection refused (error 111)
I20260929 16:00:42.230556 5544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:36347,127.23.222.125:45649,127.23.222.124:37125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab
--principal=kudu/127.23.222.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.125:45649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.222.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:42.232092 5544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:42.233489 5544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:42.246048 5550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:42.246116 5552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:42.246138 5549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:42.246963 5544 server_base.cc:1034] running on GCE node
I20260929 16:00:42.247889 5544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:42.249383 5544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:42.250612 5544 hybrid_clock.cc:648] HybridClock initialized: now 1790697642250576 us; error 54 us; skew 500 ppm
I20260929 16:00:42.251012 5544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:42.260084 5544 init.cc:377] Logged in from keytab as kudu/127.23.222.125@KRBTEST.COM (short username kudu)
I20260929 16:00:42.261825 5544 webserver.cc:460] Webserver started at http://127.23.222.125:38345/ using document root <none> and password file <none>
I20260929 16:00:42.262595 5544 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:42.262743 5544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:42.263165 5544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:00:42.266487 5544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/data/instance:
uuid: "82b81c0137d842da9ac030724ddf8271"
format_stamp: "Formatted at 2026-09-29 16:00:42 on dist-test-slave-cq88"
server_key: "b11895776a37d71f06d0d6298ce92892"
server_key_iv: "9a91a2214eb27d78955f1e40184a4ca1"
server_key_version: "encryptionkey@0"
I20260929 16:00:42.267423 5544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/wal/instance:
uuid: "82b81c0137d842da9ac030724ddf8271"
format_stamp: "Formatted at 2026-09-29 16:00:42 on dist-test-slave-cq88"
server_key: "b11895776a37d71f06d0d6298ce92892"
server_key_iv: "9a91a2214eb27d78955f1e40184a4ca1"
server_key_version: "encryptionkey@0"
I20260929 16:00:42.273069 5544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:00:42.276795 5559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:42.278244 5544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:00:42.278421 5544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/wal
uuid: "82b81c0137d842da9ac030724ddf8271"
format_stamp: "Formatted at 2026-09-29 16:00:42 on dist-test-slave-cq88"
server_key: "b11895776a37d71f06d0d6298ce92892"
server_key_iv: "9a91a2214eb27d78955f1e40184a4ca1"
server_key_version: "encryptionkey@0"
I20260929 16:00:42.278620 5544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:42.296092 5544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:42.300181 5544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:42.300457 5544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:42.319103 5544 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.125:45649
I20260929 16:00:42.319164 5610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.125:45649 every 8 connection(s)
I20260929 16:00:42.320542 5544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
I20260929 16:00:42.324221 5611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:42.325934 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 5544
I20260929 16:00:42.326113 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-1/wal/instance
I20260929 16:00:42.326715 24441 external_mini_cluster.cc:1428] Setting key 9b32bf5d401dfd352cfafc03a6c302b8
I20260929 16:00:42.334338 5611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.222.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.124@KRBTEST.COM" created.
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:42.358719 5541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.337723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.23.222.126:36347)
0929 16:00:42.337902 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.337906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.339328 (+ 1422us) server_negotiation.cc:407] Connection header received
0929 16:00:42.341318 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.341324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.341418 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.341526 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.343142 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.343874 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.345631 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.346068 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.346218 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.352525 (+ 6307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.352531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.352535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.352573 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.355354 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.356173 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.356179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.356182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.356309 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.356740 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.356749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.356752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.357174 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.357338 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.358231 (+ 893us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.358488 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:00:42.360265 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.338318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:36347 (local address 127.0.0.1:35852)
0929 16:00:42.339140 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.339187 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:42.341027 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:42.341796 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:42.341805 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:42.342338 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:42.342944 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.342962 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.344045 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.344049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.345483 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.345500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.346333 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.346341 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.346555 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.347268 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:42.347304 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:42.352258 (+ 4954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:42.355522 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.355532 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.355551 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.356038 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.356422 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.356427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.356431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.356622 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.357328 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:42.357335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:42.357803 (+ 468us) client_negotiation.cc:769] Sending connection context
0929 16:00:42.358106 (+ 303us) client_negotiation.cc:240] Negotiation successful
0929 16:00:42.358709 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":338,"threads_started":1}
I20260929 16:00:42.362210 5611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
I20260929 16:00:42.378786 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.363097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:45424)
0929 16:00:42.363330 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.363355 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:42.363472 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:42.364666 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:42.364670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:42.365055 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:42.365202 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.365211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.367353 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.367357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.368226 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.368238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.369076 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.369081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.369332 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.369913 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:42.369931 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:42.372393 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:42.375959 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.375962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.375965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.376419 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.376887 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.376891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.376893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.377039 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.377699 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:42.377703 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:42.377813 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 16:00:42.378040 (+ 227us) client_negotiation.cc:240] Negotiation successful
0929 16:00:42.378346 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":5}
I20260929 16:00:42.378830 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.363296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.23.222.125:45649)
0929 16:00:42.364097 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.364107 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.364143 (+ 36us) server_negotiation.cc:407] Connection header received
0929 16:00:42.364210 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.364221 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.364486 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.364635 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.365377 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.367157 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.368366 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.368930 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.369063 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.372613 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.372625 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.372639 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.372688 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.375809 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.376547 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.376557 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.376560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.376730 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.377129 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.377132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.377133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.377551 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.377767 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.378205 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.378452 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":598,"thread_start_us":444,"threads_started":1}
I20260929 16:00:42.381436 5611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab.
Entry for principal kudu/127.23.222.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab.
I20260929 16:00:42.382680 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.382155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46820)
0929 16:00:42.382377 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.382512 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 16:00:42.382931 5563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
I20260929 16:00:42.383478 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.124:37125
--webserver_interface=127.23.222.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab
--principal=kudu/127.23.222.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:36347,127.23.222.125:45649,127.23.222.124:37125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:42.384601 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.384239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46832)
0929 16:00:42.384417 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.384493 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 16:00:42.384827 5611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:37125: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
I20260929 16:00:42.434221 5611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } attempt: 1
I20260929 16:00:42.435603 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.434979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46836)
0929 16:00:42.435253 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.435395 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 16:00:42.436728 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.436302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46852)
0929 16:00:42.436492 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.436586 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 16:00:42.436965 5611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:37125: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
I20260929 16:00:42.505259 5611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } attempt: 2
I20260929 16:00:42.506698 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.506038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46856)
0929 16:00:42.506312 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.506450 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 16:00:42.507825 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.507416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46862)
0929 16:00:42.507588 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.507677 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:00:42.508052 5611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:37125: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
I20260929 16:00:42.522294 5537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } attempt: 5
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
I20260929 16:00:42.536387 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.523128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.23.222.125:45649)
0929 16:00:42.523343 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.523349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.523371 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:00:42.523489 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.523499 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.523562 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.523665 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.524458 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.525149 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.526349 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.526802 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.526932 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.530655 (+ 3723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.530660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.530663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.530704 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.533431 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.534302 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.534308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.534316 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.534450 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.534858 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.534863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.534871 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.535193 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.535410 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.535819 (+ 409us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.536046 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:00:42.536391 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.523002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:45432)
0929 16:00:42.523186 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.523208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:42.523357 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:42.523711 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:42.523717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:42.524114 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:42.524297 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.524306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.525296 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.525302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.526189 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.526201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.526948 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.526953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.527133 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.527860 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:42.527879 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:42.530439 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:42.533723 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.533729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.533734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.534150 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.534575 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.534580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.534584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.534741 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.535368 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:42.535373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:42.535546 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 16:00:42.535706 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 16:00:42.536030 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:00:42.537796 5537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:00:42.538908 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.538509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46878)
0929 16:00:42.538693 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.538778 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:00:42.539824 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.539491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46880)
0929 16:00:42.539657 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.539722 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:00:42.540042 5537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:37125: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
I20260929 16:00:42.591312 5537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } attempt: 1
I20260929 16:00:42.592656 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.592061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46886)
0929 16:00:42.592320 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.592445 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 16:00:42.593787 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.593381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46890)
0929 16:00:42.593553 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.593649 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:00:42.594017 5537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:37125: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
I20260929 16:00:42.599323 5611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } attempt: 3
I20260929 16:00:42.600562 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.600034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46896)
0929 16:00:42.600250 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.600369 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:00:42.601760 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.601257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46912)
0929 16:00:42.601508 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.601609 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260929 16:00:42.602033 5611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:37125: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
I20260929 16:00:42.662261 5537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } attempt: 2
I20260929 16:00:42.663777 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.663135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46918)
0929 16:00:42.663422 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.663551 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:00:42.665082 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.664639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46930)
0929 16:00:42.664819 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.664916 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 16:00:42.665366 5537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:37125: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
W20260929 16:00:42.724995 5619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:42.725306 5619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:42.725417 5619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:42.732810 5619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:42.732936 5619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:42.732983 5619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:42.733044 5619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:42.733089 5619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:42.743290 5619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:36347,127.23.222.125:45649,127.23.222.124:37125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab
--principal=kudu/127.23.222.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.124:37125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.222.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:42.744755 5619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:42.745270 5611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } attempt: 4
I20260929 16:00:42.746383 5619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 16:00:42.746526 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.745938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46940)
0929 16:00:42.746211 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.746328 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 16:00:42.747601 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.747174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46946)
0929 16:00:42.747371 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.747463 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260929 16:00:42.747839 5611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:37125: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
W20260929 16:00:42.759349 5624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:42.759532 5627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:42.759761 5625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:42.760740 5619 server_base.cc:1034] running on GCE node
I20260929 16:00:42.761612 5619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:42.763185 5619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:42.764448 5619 hybrid_clock.cc:648] HybridClock initialized: now 1790697642764397 us; error 48 us; skew 500 ppm
I20260929 16:00:42.764869 5619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:42.768491 5619 init.cc:377] Logged in from keytab as kudu/127.23.222.124@KRBTEST.COM (short username kudu)
I20260929 16:00:42.770098 5619 webserver.cc:460] Webserver started at http://127.23.222.124:45463/ using document root <none> and password file <none>
I20260929 16:00:42.770890 5619 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:42.771035 5619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:42.771440 5619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:00:42.771618 5537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } attempt: 3
I20260929 16:00:42.772815 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.772287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46950)
0929 16:00:42.772546 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.772648 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 16:00:42.773936 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.773558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46962)
0929 16:00:42.773739 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.773824 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 16:00:42.774223 5537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:37125: Network error: Client connection negotiation failed: client connection to 127.23.222.124:37125: connect: Connection refused (error 111)
I20260929 16:00:42.775436 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/data/instance:
uuid: "fdf7f6229d304851b2f05451a6fcf8b1"
format_stamp: "Formatted at 2026-09-29 16:00:42 on dist-test-slave-cq88"
server_key: "577bb4e16892d06c21bd74e62e137115"
server_key_iv: "8afed4a175df70de01aba23eae305d24"
server_key_version: "encryptionkey@0"
I20260929 16:00:42.776391 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/wal/instance:
uuid: "fdf7f6229d304851b2f05451a6fcf8b1"
format_stamp: "Formatted at 2026-09-29 16:00:42 on dist-test-slave-cq88"
server_key: "577bb4e16892d06c21bd74e62e137115"
server_key_iv: "8afed4a175df70de01aba23eae305d24"
server_key_version: "encryptionkey@0"
I20260929 16:00:42.782405 5619 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 16:00:42.786779 5634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:42.788441 5619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:00:42.788654 5619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/wal
uuid: "fdf7f6229d304851b2f05451a6fcf8b1"
format_stamp: "Formatted at 2026-09-29 16:00:42 on dist-test-slave-cq88"
server_key: "577bb4e16892d06c21bd74e62e137115"
server_key_iv: "8afed4a175df70de01aba23eae305d24"
server_key_version: "encryptionkey@0"
I20260929 16:00:42.788897 5619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:42.822419 5619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:42.827749 5619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:42.828024 5619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:42.847143 5619 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.124:37125
I20260929 16:00:42.847169 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.124:37125 every 8 connection(s)
I20260929 16:00:42.848606 5619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
I20260929 16:00:42.852392 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:42.856846 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 5619
I20260929 16:00:42.857051 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790697431711191-24441-0/minicluster-data/master-2/wal/instance
I20260929 16:00:42.857702 24441 external_mini_cluster.cc:1428] Setting key 7d519ecb42b8fa460b975ecc04395b3f
I20260929 16:00:42.860388 5686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
I20260929 16:00:42.881397 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.864555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.23.222.126:36347)
0929 16:00:42.865171 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.865177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.865697 (+ 520us) server_negotiation.cc:407] Connection header received
0929 16:00:42.865870 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.865874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.865937 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.866032 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.867462 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.868593 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.870422 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.871098 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.871345 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.874744 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.874749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.874752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.874799 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.877947 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.878785 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.878791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.878795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.878930 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.879459 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.879465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.879468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.879765 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.879934 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.880745 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.880983 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":296,"threads_started":1}
I20260929 16:00:42.882530 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.864547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:36347 (local address 127.0.0.1:35868)
0929 16:00:42.865542 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.865591 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:42.865757 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:42.866120 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:42.866127 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:42.866916 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:42.867295 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.867306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.868738 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.868741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.870286 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.870300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.871401 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.871406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.871524 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.872084 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:42.872106 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:42.874500 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:42.878104 (+ 3604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.878117 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.878158 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.878650 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.879043 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.879048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.879051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.879347 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.879894 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:42.879904 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:42.880323 (+ 419us) client_negotiation.cc:769] Sending connection context
0929 16:00:42.880609 (+ 286us) client_negotiation.cc:240] Negotiation successful
0929 16:00:42.881131 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":46,"spinlock_wait_cycles":37632,"thread_start_us":557,"threads_started":1}
I20260929 16:00:42.882551 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.859837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.23.222.124:37125)
0929 16:00:42.860695 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.860708 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.860748 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:00:42.862432 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.862457 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.862930 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.863310 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.865062 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.867281 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.869118 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.870240 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.870433 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.875301 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.875312 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.875326 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.875356 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.878455 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.879190 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.879197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.879199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.879405 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.879795 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.879800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.879804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.880303 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.880601 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.880948 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.881266 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":615,"thread_start_us":408,"threads_started":1}
I20260929 16:00:42.884480 5686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:42.898269 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.885110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:45649 (local address 127.0.0.1:45438)
0929 16:00:42.885294 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.885313 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:42.885587 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:42.886058 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:42.886066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:42.886454 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:42.886598 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.886607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.887792 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.887800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.888701 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.888713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.889420 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.889425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.889651 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.890236 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:42.890262 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:42.892546 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:42.895820 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.895825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.895830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.896227 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.896620 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.896625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.896629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.896775 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.897265 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:42.897270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:42.897435 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 16:00:42.897589 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:00:42.897890 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:00:42.898291 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.885302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.23.222.125:45649)
0929 16:00:42.885473 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.885477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.885493 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:00:42.885740 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.885748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.885833 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.885913 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.886862 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.887658 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.888843 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.889277 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.889411 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.892752 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.892758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.892761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.892797 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.895686 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.896350 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.896355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.896359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.896508 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.896882 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.896887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.896890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.897151 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.897303 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.897686 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.897908 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:00:42.899326 5686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:00:42.901136 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.886122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.23.222.126:36347)
0929 16:00:42.886314 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.886320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.886337 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:00:42.886496 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.886501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.886581 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.886679 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.887957 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.888664 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.889940 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.890558 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.890739 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.895198 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.895204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.895207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.895298 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.897865 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.898683 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.898689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.898692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.898840 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.899365 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.899370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.899372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.899746 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.900011 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.900530 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.900801 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
I20260929 16:00:42.914585 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.900232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.23.222.124:37125)
0929 16:00:42.900399 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.900405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.900494 (+ 89us) server_negotiation.cc:407] Connection header received
0929 16:00:42.900661 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.900666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.900755 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.900855 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.901730 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.902651 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.903935 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.904620 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.904771 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.908340 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.908345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.908348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.908413 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.912013 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.912658 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.912664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.912668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.912820 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.913222 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.913229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.913235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.913622 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.913779 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.914111 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.914324 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":45}
I20260929 16:00:42.915023 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.900232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46976)
0929 16:00:42.900400 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.900420 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:42.900547 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:42.900898 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:42.900903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:42.901376 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:42.901558 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.901571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.902784 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.902792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.903797 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.903809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.904762 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.904766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.904971 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.905579 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:42.905601 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:42.908111 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:42.912170 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.912174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.912177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.912530 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.912949 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.912954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.912957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.913106 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.913737 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:42.913740 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:42.913859 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 16:00:42.914016 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 16:00:42.914509 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":115}
I20260929 16:00:42.920369 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.902902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.23.222.125:45649)
0929 16:00:42.903089 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.903095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.903114 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:00:42.903218 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.903223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.903341 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.903437 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.904954 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.906014 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.907630 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.908314 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.908402 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.914027 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.914033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.914038 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.914082 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.917654 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.918382 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.918389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.918394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.918564 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.918936 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.918941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.918945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.919324 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.919575 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.919810 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.920035 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:00:42.921116 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1: Bootstrap starting.
I20260929 16:00:42.924820 5686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:42.925243 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:00:42.925962 5686 log.cc:824] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:42.929046 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1: No bootstrap required, opened a new log
I20260929 16:00:42.943346 5686 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } }
I20260929 16:00:42.944072 5686 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:42.944264 5686 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf7f6229d304851b2f05451a6fcf8b1, State: Initialized, Role: FOLLOWER
I20260929 16:00:42.945200 5686 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } }
I20260929 16:00:42.951447 5537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } attempt: 4
I20260929 16:00:42.954357 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:42.954953 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:00:42.955049 5686 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:42.962193 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.930430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.23.222.124:37125)
0929 16:00:42.930606 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.930612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.938924 (+ 8312us) server_negotiation.cc:407] Connection header received
0929 16:00:42.939136 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.939142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.939276 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.939375 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.942286 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.943445 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.945902 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.946635 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.946744 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.951544 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.951551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.951555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.951601 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.958536 (+ 6935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.959509 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.959516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.959523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.959710 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.960187 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.960195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.960201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.960644 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.960895 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.961550 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.961795 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 16:00:42 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697641, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
I20260929 16:00:42.966929 5617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.930875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.23.222.125:45649)
0929 16:00:42.931039 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.931046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.937993 (+ 6947us) server_negotiation.cc:407] Connection header received
0929 16:00:42.939357 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.939362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.939446 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.939516 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.942286 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.943469 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.946083 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.946744 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.946835 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.951551 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.951557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.951561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.951605 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.956660 (+ 5055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.964169 (+ 7509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.964177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.964181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.964379 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.964887 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.964895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.964902 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.965338 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.965582 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.966180 (+ 598us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.966439 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:00:42.969970 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.930370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.23.222.126:36347)
0929 16:00:42.930581 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.930587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.942966 (+ 12379us) server_negotiation.cc:407] Connection header received
0929 16:00:42.943464 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.943470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.943558 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.943631 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.946415 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.947872 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.950994 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.951788 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.951898 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.959259 (+ 7361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.959268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.959277 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.959332 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.963627 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.967462 (+ 3835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.967468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.967472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.967616 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.968123 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.968129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.968132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.968475 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.968724 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.969333 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.969560 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:00:42.976624 5540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.952036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46994)
0929 16:00:42.952324 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:42.952344 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:42.952441 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:42.956874 (+ 4433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:42.956881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:42.957502 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:42.957756 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.957773 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.960140 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.960149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.962266 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:42.962290 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.964057 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.964065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:42.964332 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.965000 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:42.965024 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:42.967748 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:42.972349 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.972356 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.972363 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.973338 (+ 975us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.973904 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:42.973913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:42.973917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:42.974122 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:42.975475 (+ 1353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:42.975483 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:42.975682 (+ 199us) client_negotiation.cc:769] Sending connection context
0929 16:00:42.975848 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 16:00:42.976177 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260929 16:00:42.977620 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:42.953084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46994 (local address 127.23.222.124:37125)
0929 16:00:42.956585 (+ 3501us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:42.956590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:42.956609 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:00:42.956657 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:42.956660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:42.956715 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:42.956830 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:42.957923 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.959936 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.962845 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:42.963843 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:42.964717 (+ 874us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:42.967973 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:42.967981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:42.967988 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:42.968027 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:42.972145 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.973555 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.973563 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.973568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.973745 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:42.974266 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:42.974273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:42.974279 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:42.974757 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:42.976874 (+ 2117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:42.977019 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 16:00:42.977263 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3357,"thread_start_us":3129,"threads_started":1}
W20260929 16:00:42.979362 5712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:00:42.979609 5712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:00:42.983053 5537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48: Bootstrap starting.
I20260929 16:00:42.974818 5619 master_runner.cc:186] Error getting master registration for 127.23.222.126:36347: OK, instance_id { permanent_uuid: "001927311b244371a09c0a87e2849f48" instance_seqno: 1790697641807501 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:00:42.985041 5619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:00:42.986363 5537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:42.987739 5537 log.cc:824] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:42.993989 5537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48: No bootstrap required, opened a new log
I20260929 16:00:43.002377 5537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } }
I20260929 16:00:43.002709 5537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:43.002859 5537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001927311b244371a09c0a87e2849f48, State: Initialized, Role: FOLLOWER
I20260929 16:00:43.003479 5537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } }
I20260929 16:00:43.004418 5715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:43.004799 5715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:00:43.005628 5537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:00:43.017571 5726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:00:43.017724 5726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:00:43.015255 5469 master_runner.cc:186] Error getting master registration for 127.23.222.125:45649: OK, instance_id { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" instance_seqno: 1790697642313750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:00:43.025349 5469 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 16:00:43.047101 5491 tablet.cc:2369] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:00:43.047338 5611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } attempt: 5
Sep 29 16:00:43 dist-test-slave-cq88 krb5kdc[5456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697642, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
I20260929 16:00:43.066320 5614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:43.047995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:37125 (local address 127.0.0.1:46998)
0929 16:00:43.048376 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:43.048397 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:43.048510 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:43.049478 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:43.049486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:43.050037 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:43.050262 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:43.050275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:43.051788 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:43.051794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:43.053269 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:43.053286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:43.054299 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:43.054308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:43.054550 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:43.055347 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:43.055371 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:43.058250 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:43.062769 (+ 4519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:43.062775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:43.062780 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:43.063349 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:43.063823 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:43.063828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:43.063832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:43.064034 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:43.064795 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:43.064800 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:43.064985 (+ 185us) client_negotiation.cc:769] Sending connection context
0929 16:00:43.065589 (+ 604us) client_negotiation.cc:240] Negotiation successful
0929 16:00:43.065910 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250}
I20260929 16:00:43.067279 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:43.048755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.23.222.124:37125)
0929 16:00:43.048932 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:43.048938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:43.048955 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:00:43.049014 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:43.049019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:43.049120 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:43.049223 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:43.050444 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:43.051614 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:43.053422 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:43.054125 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:43.054272 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:43.058497 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:43.058506 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:43.058513 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:43.058557 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:43.062597 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:43.063478 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:43.063485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:43.063489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:43.063692 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:43.064189 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:43.064195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:43.064198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:43.064557 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:43.064725 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:43.066623 (+ 1898us) server_negotiation.cc:299] Negotiation successful
0929 16:00:43.066876 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:00:43.073485 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271: Bootstrap starting.
I20260929 16:00:43.077641 5611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:43.078969 5611 log.cc:824] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271: Log is configured to *not* fsync() on all Append() calls
W20260929 16:00:43.080267 5640 tablet.cc:2369] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:00:43.084581 5611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271: No bootstrap required, opened a new log
I20260929 16:00:43.095866 5611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } }
I20260929 16:00:43.096320 5611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:43.096434 5611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82b81c0137d842da9ac030724ddf8271, State: Initialized, Role: FOLLOWER
I20260929 16:00:43.097241 5611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } }
I20260929 16:00:43.100268 5729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:43.100735 5729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:00:43.101711 5611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:43.118834 5544 master_runner.cc:263] No leader master found from master 82b81c0137d842da9ac030724ddf8271
I20260929 16:00:43.119002 5544 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 16:00:43.122118 5740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:00:43.122366 5740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:00:43.173250 5729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:00:43.173662 5729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } }
I20260929 16:00:43.176024 5729 leader_election.cc:288] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 001927311b244371a09c0a87e2849f48 (127.23.222.126:36347), fdf7f6229d304851b2f05451a6fcf8b1 (127.23.222.124:37125)
I20260929 16:00:43.178925 5661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b81c0137d842da9ac030724ddf8271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" is_pre_election: true
I20260929 16:00:43.179709 5661 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82b81c0137d842da9ac030724ddf8271 in term 0.
I20260929 16:00:43.180729 5563 leader_election.cc:302] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82b81c0137d842da9ac030724ddf8271, fdf7f6229d304851b2f05451a6fcf8b1; no voters:
I20260929 16:00:43.177503 5512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b81c0137d842da9ac030724ddf8271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001927311b244371a09c0a87e2849f48" is_pre_election: true
I20260929 16:00:43.182019 5512 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82b81c0137d842da9ac030724ddf8271 in term 0.
I20260929 16:00:43.188709 5729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:00:43.189034 5729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:00:43.189218 5729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:43.193317 5729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } }
I20260929 16:00:43.195048 5512 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b81c0137d842da9ac030724ddf8271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001927311b244371a09c0a87e2849f48"
I20260929 16:00:43.195353 5512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:43.199072 5512 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82b81c0137d842da9ac030724ddf8271 in term 1.
I20260929 16:00:43.199872 5562 leader_election.cc:302] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001927311b244371a09c0a87e2849f48, 82b81c0137d842da9ac030724ddf8271; no voters:
I20260929 16:00:43.205300 5661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82b81c0137d842da9ac030724ddf8271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdf7f6229d304851b2f05451a6fcf8b1"
I20260929 16:00:43.205618 5661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:43.209582 5661 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82b81c0137d842da9ac030724ddf8271 in term 1.
I20260929 16:00:43.216145 5729 leader_election.cc:288] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [CANDIDATE]: Term 1 election: Requested vote from peers 001927311b244371a09c0a87e2849f48 (127.23.222.126:36347), fdf7f6229d304851b2f05451a6fcf8b1 (127.23.222.124:37125)
I20260929 16:00:43.216620 5729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:00:43.217056 5729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [term 1 LEADER]: Becoming Leader. State: Replica: 82b81c0137d842da9ac030724ddf8271, State: Running, Role: LEADER
I20260929 16:00:43.217844 5729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } }
I20260929 16:00:43.236630 5742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b81c0137d842da9ac030724ddf8271. Latest consensus state: current_term: 1 leader_uuid: "82b81c0137d842da9ac030724ddf8271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:43.237215 5742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:43.250192 5744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:00:43.252612 5744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:00:43.272742 5661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [term 1 FOLLOWER]: Refusing update from remote peer 82b81c0137d842da9ac030724ddf8271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:43.274034 5742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:00:43.281705 5512 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [term 1 FOLLOWER]: Refusing update from remote peer 82b81c0137d842da9ac030724ddf8271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:43.282915 5742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 16:00:43.303086 5565 tablet.cc:2369] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:00:43.306142 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b81c0137d842da9ac030724ddf8271. Latest consensus state: current_term: 1 leader_uuid: "82b81c0137d842da9ac030724ddf8271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:43.306491 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:00:43.314142 5746 mvcc.cc:205] Tried to move back new op lower bound from 7334697546833240064 to 7334697546633273344. Current Snapshot: MvccSnapshot[applied={T|T < 7334697546833240064}]
I20260929 16:00:43.315634 5745 mvcc.cc:205] Tried to move back new op lower bound from 7334697546833240064 to 7334697546633273344. Current Snapshot: MvccSnapshot[applied={T|T < 7334697546833240064}]
I20260929 16:00:43.325289 5729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82b81c0137d842da9ac030724ddf8271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:43.325642 5729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:43.327697 5700 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82b81c0137d842da9ac030724ddf8271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:43.328032 5700 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:00:43.329587 5744 catalog_manager.cc:1347] Generated new cluster ID: 8544d913d39b4f5a86979ddc9f805f74
I20260929 16:00:43.329689 5744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:00:43.345510 5715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82b81c0137d842da9ac030724ddf8271. Latest consensus state: current_term: 1 leader_uuid: "82b81c0137d842da9ac030724ddf8271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:43.345889 5715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:00:43.349371 5742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82b81c0137d842da9ac030724ddf8271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:43.349658 5742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271 [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:43.359735 5715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82b81c0137d842da9ac030724ddf8271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001927311b244371a09c0a87e2849f48" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 36347 } } peers { permanent_uuid: "82b81c0137d842da9ac030724ddf8271" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 45649 } } peers { permanent_uuid: "fdf7f6229d304851b2f05451a6fcf8b1" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 37125 } } }
I20260929 16:00:43.360086 5715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:00:43.376363 5744 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:00:43.378789 5744 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:00:43.404737 5744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82b81c0137d842da9ac030724ddf8271: Generated new TSK 0
I20260929 16:00:43.406219 5744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:00:43.981704 5712 catalog_manager.cc:1259] Loaded cluster ID: 8544d913d39b4f5a86979ddc9f805f74
I20260929 16:00:43.981864 5712 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1: loading cluster ID for follower catalog manager: success
I20260929 16:00:43.984401 5712 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1: acquiring CA information for follower catalog manager: success
I20260929 16:00:43.986158 5712 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fdf7f6229d304851b2f05451a6fcf8b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:00:44.020258 5726 catalog_manager.cc:1259] Loaded cluster ID: 8544d913d39b4f5a86979ddc9f805f74
I20260929 16:00:44.020383 5726 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48: loading cluster ID for follower catalog manager: success
I20260929 16:00:44.022904 5726 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48: acquiring CA information for follower catalog manager: success
I20260929 16:00:44.024716 5726 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 001927311b244371a09c0a87e2849f48: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:00:44.537640 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 5469
I20260929 16:00:44.539026 5532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:44.649335 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 5469
I20260929 16:00:44.670661 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 5544
I20260929 16:00:44.671903 5606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:44.780532 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 5544
I20260929 16:00:44.801631 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 5619
I20260929 16:00:44.802863 5681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:44.910293 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 5619
2026-09-29T16:00:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4888 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:00:45 dist-test-slave-cq88 krb5kdc[5762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:00:45 dist-test-slave-cq88 krb5kdc[5762](info): set up 2 sockets
Sep 29 16:00:45 dist-test-slave-cq88 krb5kdc[5762](info): commencing operation
krb5kdc: starting...
W20260929 16:00:46.006878 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:00:46 dist-test-slave-cq88 krb5kdc[5762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:00:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:00:46.259351 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38681
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:38681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:46.602653 5775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:46.603009 5775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:46.603188 5775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:46.610921 5775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:46.611063 5775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:46.611192 5775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:46.611349 5775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:46.611436 5775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:46.622251 5775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:38681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:46.623924 5775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:46.625416 5775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:46.638471 5780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:46.638602 5781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:46.639101 5783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:46.639648 5775 server_base.cc:1034] running on GCE node
I20260929 16:00:46.640532 5775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:46.642023 5775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:46.643335 5775 hybrid_clock.cc:648] HybridClock initialized: now 1790697646643211 us; error 43 us; skew 500 ppm
I20260929 16:00:46.643778 5775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:46 dist-test-slave-cq88 krb5kdc[5762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697646, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:46.647145 5775 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:00:46.648825 5775 webserver.cc:460] Webserver started at http://127.23.222.126:46685/ using document root <none> and password file <none>
I20260929 16:00:46.649561 5775 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:46.649703 5775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:46.650029 5775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:00:46.653225 5775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "054fb1299319419d936d2eca7b8d4181"
format_stamp: "Formatted at 2026-09-29 16:00:46 on dist-test-slave-cq88"
server_key: "3d571592c676192ee18fbe52b7b6d218"
server_key_iv: "97dc217cfe37162628b2fc00c7dff055"
server_key_version: "encryptionkey@0"
I20260929 16:00:46.654008 5775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "054fb1299319419d936d2eca7b8d4181"
format_stamp: "Formatted at 2026-09-29 16:00:46 on dist-test-slave-cq88"
server_key: "3d571592c676192ee18fbe52b7b6d218"
server_key_iv: "97dc217cfe37162628b2fc00c7dff055"
server_key_version: "encryptionkey@0"
I20260929 16:00:46.659520 5775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 16:00:46.663200 5790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:46.664793 5775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:00:46.664999 5775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "054fb1299319419d936d2eca7b8d4181"
format_stamp: "Formatted at 2026-09-29 16:00:46 on dist-test-slave-cq88"
server_key: "3d571592c676192ee18fbe52b7b6d218"
server_key_iv: "97dc217cfe37162628b2fc00c7dff055"
server_key_version: "encryptionkey@0"
I20260929 16:00:46.665194 5775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:46.707568 5775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:46.712308 5775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:46.712663 5775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:46.731745 5775 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:38681
I20260929 16:00:46.731765 5841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:38681 every 8 connection(s)
I20260929 16:00:46.733204 5775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:00:46.733711 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 5775
I20260929 16:00:46.733954 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:00:46.734839 24441 external_mini_cluster.cc:1428] Setting key 177d3fb8ec5c3304cba594789d9cf832
I20260929 16:00:46.738147 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:00:46 dist-test-slave-cq88 krb5kdc[5762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:46.752876 5842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181: Bootstrap starting.
I20260929 16:00:46.757750 5842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:46.759105 5842 log.cc:824] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:46.763019 5842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181: No bootstrap required, opened a new log
I20260929 16:00:46.763276 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:46.736831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.23.222.126:38681)
0929 16:00:46.737758 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:46.737775 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:46.737830 (+ 55us) server_negotiation.cc:407] Connection header received
0929 16:00:46.740295 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:46.740327 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:46.740875 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:46.741360 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:46.742827 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:46.745434 (+ 2607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:46.747099 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:46.748276 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:46.748483 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:46.753517 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:46.753530 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:46.753554 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:46.753627 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:46.758055 (+ 4428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:46.759042 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:46.759054 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:46.759063 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:46.759344 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:46.759840 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:46.759848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:46.759854 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:46.760495 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:46.760848 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:46.761287 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 16:00:46.761800 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":374,"threads_started":1}
I20260929 16:00:46.768105 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:00:46.773212 5842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054fb1299319419d936d2eca7b8d4181" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38681 } }
I20260929 16:00:46.773635 5842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:46.773758 5842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 054fb1299319419d936d2eca7b8d4181, State: Initialized, Role: FOLLOWER
I20260929 16:00:46.774435 5842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054fb1299319419d936d2eca7b8d4181" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38681 } }
I20260929 16:00:46.774667 5842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:00:46.774827 5842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:00:46.775009 5842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:46.777356 5842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054fb1299319419d936d2eca7b8d4181" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38681 } }
I20260929 16:00:46.777944 5842 leader_election.cc:302] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 054fb1299319419d936d2eca7b8d4181; no voters:
I20260929 16:00:46.778760 5842 leader_election.cc:288] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:00:46.778966 5847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:00:46.779764 5847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [term 1 LEADER]: Becoming Leader. State: Replica: 054fb1299319419d936d2eca7b8d4181, State: Running, Role: LEADER
I20260929 16:00:46.780352 5847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054fb1299319419d936d2eca7b8d4181" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38681 } }
I20260929 16:00:46.781306 5842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:46.783519 5848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "054fb1299319419d936d2eca7b8d4181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054fb1299319419d936d2eca7b8d4181" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38681 } } }
I20260929 16:00:46.783869 5848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:46.784329 5849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 054fb1299319419d936d2eca7b8d4181. Latest consensus state: current_term: 1 leader_uuid: "054fb1299319419d936d2eca7b8d4181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "054fb1299319419d936d2eca7b8d4181" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38681 } } }
I20260929 16:00:46.784901 5849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181 [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:46.786263 5854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:00:46.792263 5854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:00:46.803371 5854 catalog_manager.cc:1347] Generated new cluster ID: 1504d46d2e4b47b79c98c51207f4799c
I20260929 16:00:46.803516 5854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:00:46.815881 5854 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:00:46.817312 5854 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:00:46.824090 5854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 054fb1299319419d936d2eca7b8d4181: Generated new TSK 0
I20260929 16:00:46.824957 5854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:00:46.881219 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 5775
I20260929 16:00:46.882515 5837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:46.980428 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 5775
2026-09-29T16:00:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2069 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:00:47 dist-test-slave-cq88 krb5kdc[5870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:00:47 dist-test-slave-cq88 krb5kdc[5870](info): set up 2 sockets
Sep 29 16:00:47 dist-test-slave-cq88 krb5kdc[5870](info): commencing operation
krb5kdc: starting...
W20260929 16:00:48.078930 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:00:48 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:00:48.330713 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42523
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:48.677840 5883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:48.678179 5883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:48.678406 5883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:48.686131 5883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:48.686271 5883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:48.686398 5883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:48.686483 5883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:48.686580 5883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:48.697175 5883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:42523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:48.698714 5883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:48.700309 5883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:48.713231 5888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:48.713162 5889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:48.713235 5891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:48.714299 5883 server_base.cc:1034] running on GCE node
I20260929 16:00:48.715215 5883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:48.716873 5883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:48.718144 5883 hybrid_clock.cc:648] HybridClock initialized: now 1790697648718089 us; error 52 us; skew 500 ppm
I20260929 16:00:48.718516 5883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:48 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697648, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:48.728382 5883 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:00:48.730072 5883 webserver.cc:460] Webserver started at http://127.23.222.126:43369/ using document root <none> and password file <none>
I20260929 16:00:48.730836 5883 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:48.730991 5883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:48.731382 5883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:00:48.734730 5883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "29e858a095104ac4af470dba76519e8f"
format_stamp: "Formatted at 2026-09-29 16:00:48 on dist-test-slave-cq88"
server_key: "3d8e79b2f2efa33da8baa735e9320e3f"
server_key_iv: "cc886e18dfced2f30121963fcc4bba78"
server_key_version: "encryptionkey@0"
I20260929 16:00:48.735636 5883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "29e858a095104ac4af470dba76519e8f"
format_stamp: "Formatted at 2026-09-29 16:00:48 on dist-test-slave-cq88"
server_key: "3d8e79b2f2efa33da8baa735e9320e3f"
server_key_iv: "cc886e18dfced2f30121963fcc4bba78"
server_key_version: "encryptionkey@0"
I20260929 16:00:48.741333 5883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:00:48.745231 5898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:48.746850 5883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:00:48.747077 5883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "29e858a095104ac4af470dba76519e8f"
format_stamp: "Formatted at 2026-09-29 16:00:48 on dist-test-slave-cq88"
server_key: "3d8e79b2f2efa33da8baa735e9320e3f"
server_key_iv: "cc886e18dfced2f30121963fcc4bba78"
server_key_version: "encryptionkey@0"
I20260929 16:00:48.747356 5883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:48.793152 5883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:48.797811 5883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:48.798112 5883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:48.817814 5883 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:42523
I20260929 16:00:48.817847 5951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:42523 every 8 connection(s)
I20260929 16:00:48.819366 5883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:00:48.823603 5952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:48.825282 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 5883
I20260929 16:00:48.825505 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:00:48.826208 24441 external_mini_cluster.cc:1428] Setting key 17a45398d8c5891782908d1fc3182415
I20260929 16:00:48.838804 5952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Bootstrap starting.
Sep 29 16:00:48 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:48.842864 5952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:48.844228 5952 log.cc:824] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:48.847429 5952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: No bootstrap required, opened a new log
I20260929 16:00:48.850561 5955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:48.828361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.23.222.126:42523)
0929 16:00:48.829268 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:48.829281 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:48.829326 (+ 45us) server_negotiation.cc:407] Connection header received
0929 16:00:48.831024 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:48.831050 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:48.831568 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:48.831925 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:48.833746 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:48.835075 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:48.836974 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:48.837671 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:48.837813 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:48.842816 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:48.842824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:48.842839 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:48.842889 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:48.846212 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:48.847009 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:48.847016 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:48.847022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:48.847206 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:48.847659 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:48.847670 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:48.847673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:48.848229 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:48.848565 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:48.849036 (+ 471us) server_negotiation.cc:299] Negotiation successful
0929 16:00:48.849492 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":434,"threads_started":1}
I20260929 16:00:48.856063 5952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:48.856483 5952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:48.856600 5952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e858a095104ac4af470dba76519e8f, State: Initialized, Role: FOLLOWER
I20260929 16:00:48.857291 5952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:48.857630 5952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:00:48.857825 5952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:00:48.858072 5952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:48.860770 5952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:48.861344 5952 leader_election.cc:302] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29e858a095104ac4af470dba76519e8f; no voters:
I20260929 16:00:48.861972 5952 leader_election.cc:288] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:00:48.862112 5957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:00:48.862809 5957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 1 LEADER]: Becoming Leader. State: Replica: 29e858a095104ac4af470dba76519e8f, State: Running, Role: LEADER
I20260929 16:00:48.863471 5957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:48.864135 5952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:48.866262 5959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29e858a095104ac4af470dba76519e8f. Latest consensus state: current_term: 1 leader_uuid: "29e858a095104ac4af470dba76519e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } } }
I20260929 16:00:48.866576 5959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:48.867496 5964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:00:48.867327 5958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29e858a095104ac4af470dba76519e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } } }
I20260929 16:00:48.872839 5958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:48.875074 5964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:00:48.887809 5964 catalog_manager.cc:1347] Generated new cluster ID: 0f39dd2378e246c5a46967de256d16bf
I20260929 16:00:48.888288 5964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:00:48.904202 5964 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:00:48.905622 5964 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:00:48.923461 5964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Generated new TSK 0
I20260929 16:00:48.924407 5964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 16:00:48.995981 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42523
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:49.338549 5978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:49.338882 5978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:49.339058 5978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:49.346589 5978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:49.346751 5978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:49.346942 5978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:00:49.357827 5978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:49.359287 5978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:49.360721 5978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:49.374207 5983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:49.374238 5986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:49.374285 5984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:49.375272 5978 server_base.cc:1034] running on GCE node
I20260929 16:00:49.375943 5978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:49.376962 5978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:49.378203 5978 hybrid_clock.cc:648] HybridClock initialized: now 1790697649378153 us; error 59 us; skew 500 ppm
I20260929 16:00:49.378544 5978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:49 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697649, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:49.381902 5978 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:00:49.383437 5978 webserver.cc:460] Webserver started at http://127.23.222.65:34547/ using document root <none> and password file <none>
I20260929 16:00:49.384155 5978 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:49.384298 5978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:49.384616 5978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:00:49.387656 5978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "c22709258bbf47e9b5c50ddf62c3702c"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "842789e5f226299a63c44eb1176fef07"
server_key_iv: "3d2260844205cb813e3d6a6c0ccbbca7"
server_key_version: "encryptionkey@0"
I20260929 16:00:49.388442 5978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "c22709258bbf47e9b5c50ddf62c3702c"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "842789e5f226299a63c44eb1176fef07"
server_key_iv: "3d2260844205cb813e3d6a6c0ccbbca7"
server_key_version: "encryptionkey@0"
I20260929 16:00:49.393826 5978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 16:00:49.397446 5993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:49.398916 5978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:00:49.399106 5978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "c22709258bbf47e9b5c50ddf62c3702c"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "842789e5f226299a63c44eb1176fef07"
server_key_iv: "3d2260844205cb813e3d6a6c0ccbbca7"
server_key_version: "encryptionkey@0"
I20260929 16:00:49.399385 5978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:49.423656 5978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:49.428455 5978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:49.428802 5978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:49.429913 5978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:49.431471 5978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:49.431594 5978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:49.431749 5978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:49.431825 5978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:49.469429 5978 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:36375
I20260929 16:00:49.469525 6107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:36375 every 8 connection(s)
I20260929 16:00:49.470940 5978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:00:49.473857 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 5978
I20260929 16:00:49.474095 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 16:00:49.474807 24441 external_mini_cluster.cc:1428] Setting key ae0da3cfd80c03b049ee649b3d45c52d
Sep 29 16:00:49 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697649, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:49.494510 6110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:49.473724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:58727 (local address 127.23.222.126:42523)
0929 16:00:49.474258 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:49.474266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:49.477394 (+ 3128us) server_negotiation.cc:407] Connection header received
0929 16:00:49.477455 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:49.477459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:49.477560 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:49.477651 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:49.479851 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:49.480682 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:49.483212 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:49.483741 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:49.483851 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:49.487407 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:49.487417 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:49.487425 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:49.487472 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:49.491079 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:49.491887 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:49.491896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:49.491902 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:49.492069 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:49.492473 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:49.492481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:49.492487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:49.492859 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:49.493014 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:49.493710 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 16:00:49.493996 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":256,"threads_started":1}
I20260929 16:00:49.495137 6111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:49.474306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:58727)
0929 16:00:49.475189 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:49.475260 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:49.477101 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:49.477957 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:49.477972 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:49.478807 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:49.479656 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:49.479688 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:49.480841 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:49.480850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:49.483028 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:49.483046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:49.483974 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:49.483984 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:49.484339 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:49.485042 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:49.485064 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:49.487145 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:49.491295 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:49.491303 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:49.491316 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:49.491751 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:49.492177 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:49.492180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:49.492182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:49.492359 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:49.492981 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:49.492987 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:49.493320 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 16:00:49.493543 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 16:00:49.493937 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":371,"threads_started":1}
I20260929 16:00:49.496819 6108 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42523
I20260929 16:00:49.497192 6108 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:49.498142 6108 heartbeater.cc:511] Master 127.23.222.126:42523 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 16:00:49.500475 5915 ts_manager.cc:194] Registered new tserver with Master: c22709258bbf47e9b5c50ddf62c3702c (127.23.222.65:36375)
I20260929 16:00:49.502074 5915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:58727
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 16:00:49.530933 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42523
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:49.864969 6114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:49.865347 6114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:49.865512 6114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:49.873108 6114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:49.873229 6114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:49.873345 6114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:00:49.883186 6114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:49.884518 6114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:49.885774 6114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:49.899147 6119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:49.899085 6120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:49.899165 6122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:49.900125 6114 server_base.cc:1034] running on GCE node
I20260929 16:00:49.900763 6114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:49.901700 6114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:49.902952 6114 hybrid_clock.cc:648] HybridClock initialized: now 1790697649902882 us; error 48 us; skew 500 ppm
I20260929 16:00:49.903373 6114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:49 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697649, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:49.906667 6114 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:00:49.908190 6114 webserver.cc:460] Webserver started at http://127.23.222.66:43019/ using document root <none> and password file <none>
I20260929 16:00:49.908905 6114 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:49.909073 6114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:49.909420 6114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:00:49.912675 6114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "436d07338a374dfcb504c48028129c58"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "844b7a2772122ea52c4e4302236b21e0"
server_key_iv: "e12d93b0c3ccf06b39e976ad378879e9"
server_key_version: "encryptionkey@0"
I20260929 16:00:49.913440 6114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "436d07338a374dfcb504c48028129c58"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "844b7a2772122ea52c4e4302236b21e0"
server_key_iv: "e12d93b0c3ccf06b39e976ad378879e9"
server_key_version: "encryptionkey@0"
I20260929 16:00:49.919142 6114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:00:49.922746 6129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:49.924283 6114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:00:49.924459 6114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "436d07338a374dfcb504c48028129c58"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "844b7a2772122ea52c4e4302236b21e0"
server_key_iv: "e12d93b0c3ccf06b39e976ad378879e9"
server_key_version: "encryptionkey@0"
I20260929 16:00:49.924675 6114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:49.967530 6114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:49.972859 6114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:49.973209 6114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:49.974237 6114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:49.975894 6114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:49.975978 6114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:49.976126 6114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:49.976189 6114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:50.012076 6114 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:42811
I20260929 16:00:50.012080 6242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:42811 every 8 connection(s)
I20260929 16:00:50.013470 6114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:00:50.015878 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 6114
I20260929 16:00:50.016078 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 16:00:50.016651 24441 external_mini_cluster.cc:1428] Setting key ae61500d5838048f0664692809410bca
Sep 29 16:00:50 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697649, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:50.038101 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.016351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:50025 (local address 127.23.222.126:42523)
0929 16:00:50.016868 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.016873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.018000 (+ 1127us) server_negotiation.cc:407] Connection header received
0929 16:00:50.019913 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.019929 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.020021 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.020145 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:50.021747 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.022984 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.024678 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.025438 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.025594 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.030237 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:50.030247 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:50.030255 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:50.030306 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:50.034249 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:50.035153 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:50.035162 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:50.035168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:50.035355 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:50.035879 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:50.035887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:50.035893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:50.036305 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:50.036470 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.037308 (+ 838us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.037578 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":260,"threads_started":1}
I20260929 16:00:50.038955 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.016949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:50025)
0929 16:00:50.017825 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:50.017878 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:50.019637 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:50.020356 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:50.020365 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:50.020923 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:50.021554 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.021571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.023152 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.023157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.024520 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.024533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.025689 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.025695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.026073 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.026963 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:50.026997 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:50.029960 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:50.034455 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:50.034469 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:50.034484 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:50.034992 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:50.035508 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:50.035512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:50.035514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:50.035734 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:50.036469 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:50.036476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:50.036872 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 16:00:50.037144 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 16:00:50.037591 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":376,"threads_started":1}
I20260929 16:00:50.040664 6243 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42523
I20260929 16:00:50.040992 6243 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:50.041785 6243 heartbeater.cc:511] Master 127.23.222.126:42523 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
I20260929 16:00:50.043483 5915 ts_manager.cc:194] Registered new tserver with Master: 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811)
I20260929 16:00:50.044433 5915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:50025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 16:00:50.074463 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42523
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:50.405298 6249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:50.405575 6249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:50.405671 6249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:50.413247 6249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:50.413357 6249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:50.413470 6249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:00:50.423060 6249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:50.424413 6249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:50.425693 6249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:50.439447 6257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:50.439522 6255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:50.439502 6254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:50.440352 6249 server_base.cc:1034] running on GCE node
I20260929 16:00:50.441148 6249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:50.442175 6249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:50.443397 6249 hybrid_clock.cc:648] HybridClock initialized: now 1790697650443344 us; error 45 us; skew 500 ppm
I20260929 16:00:50.443734 6249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:50 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697650, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:50.446954 6249 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:00:50.448521 6249 webserver.cc:460] Webserver started at http://127.23.222.67:36071/ using document root <none> and password file <none>
I20260929 16:00:50.449224 6249 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:50.449373 6249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:50.449678 6249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:00:50.452885 6249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "19f6ea2be6cb4dc499590ce2b5c8a896"
format_stamp: "Formatted at 2026-09-29 16:00:50 on dist-test-slave-cq88"
server_key: "257d5901640b32873a6b4eda331c80cc"
server_key_iv: "4404c6b7265617ecdc8f6a15dbdb5816"
server_key_version: "encryptionkey@0"
I20260929 16:00:50.453683 6249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "19f6ea2be6cb4dc499590ce2b5c8a896"
format_stamp: "Formatted at 2026-09-29 16:00:50 on dist-test-slave-cq88"
server_key: "257d5901640b32873a6b4eda331c80cc"
server_key_iv: "4404c6b7265617ecdc8f6a15dbdb5816"
server_key_version: "encryptionkey@0"
I20260929 16:00:50.459167 6249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:00:50.463089 6264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:50.464740 6249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:00:50.464917 6249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "19f6ea2be6cb4dc499590ce2b5c8a896"
format_stamp: "Formatted at 2026-09-29 16:00:50 on dist-test-slave-cq88"
server_key: "257d5901640b32873a6b4eda331c80cc"
server_key_iv: "4404c6b7265617ecdc8f6a15dbdb5816"
server_key_version: "encryptionkey@0"
I20260929 16:00:50.465191 6249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:50.481837 6249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:50.486307 6249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:50.486606 6249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:50.487664 6249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:50.489037 6249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:50.489151 6249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:50.489296 6249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:50.489373 6249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:50.505318 6108 heartbeater.cc:503] Master 127.23.222.126:42523 was elected leader, sending a full tablet report...
I20260929 16:00:50.525683 6249 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:34437
I20260929 16:00:50.525724 6376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:34437 every 8 connection(s)
I20260929 16:00:50.527005 6249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:00:50.528017 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 6249
I20260929 16:00:50.528214 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 16:00:50.528800 24441 external_mini_cluster.cc:1428] Setting key 0f57732b4e2118ad104164f01936aae6
Sep 29 16:00:50 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697650, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:50.555078 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.530015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:44419 (local address 127.23.222.126:42523)
0929 16:00:50.530188 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.530193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.531759 (+ 1566us) server_negotiation.cc:407] Connection header received
0929 16:00:50.533875 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.533882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.533989 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.534117 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:50.535903 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.536718 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.538409 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.538833 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.538993 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.549047 (+ 10054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:50.549052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:50.549055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:50.549091 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:50.551879 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:50.552703 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:50.552709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:50.552714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:50.552845 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:50.553305 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:50.553309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:50.553311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:50.553579 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:50.553764 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.554492 (+ 728us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.554731 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 16:00:50.556411 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.530701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:44419)
0929 16:00:50.531607 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:50.531654 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:50.533585 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:50.534402 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:50.534411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:50.534962 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:50.535719 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.535740 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.536892 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.536899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.538240 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.538253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.539086 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.539096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.539427 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.546416 (+ 6989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:50.546445 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:50.548805 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:50.552051 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:50.552061 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:50.552082 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:50.552566 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:50.552962 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:50.552967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:50.552971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:50.553175 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:50.553722 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:50.553731 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:50.554117 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 16:00:50.554397 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 16:00:50.554981 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":390,"threads_started":1}
I20260929 16:00:50.558207 6377 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42523
I20260929 16:00:50.558502 6377 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:50.559336 6377 heartbeater.cc:511] Master 127.23.222.126:42523 requested a full tablet report, sending...
I20260929 16:00:50.560781 5915 ts_manager.cc:194] Registered new tserver with Master: 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437)
I20260929 16:00:50.561586 5915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:44419
I20260929 16:00:50.570847 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:00:50.589166 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.575383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.23.222.126:42523)
0929 16:00:50.575613 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.575623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.576106 (+ 483us) server_negotiation.cc:407] Connection header received
0929 16:00:50.576290 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.576294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.576365 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.576458 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:50.577955 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.578717 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.580544 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.581000 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.581110 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.583702 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:50.583708 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:50.583711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:50.583749 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:50.586308 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:50.587014 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:50.587018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:50.587020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:50.587140 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:50.587614 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:50.587619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:50.587621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:50.587987 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:50.588232 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.588532 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.588738 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:00:50.593787 5915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35352:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:00:50.600696 5915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:00:50.637125 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.614041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.23.222.66:42811)
0929 16:00:50.625748 (+ 11707us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.625760 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.627304 (+ 1544us) server_negotiation.cc:407] Connection header received
0929 16:00:50.627649 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.627661 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.628142 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.628392 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:50.629947 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.631549 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.634407 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.635829 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.635984 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.636214 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.636572 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.636780 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11476,"thread_start_us":11742,"threads_started":1}
I20260929 16:00:50.637157 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.613950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:42811 (local address 127.0.0.1:52340)
0929 16:00:50.626678 (+ 12728us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:50.626721 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:50.626974 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:50.628778 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:50.628791 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:50.628937 (+ 146us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:50.629674 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.629695 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.631689 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.631697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.634167 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.634181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.636067 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.636073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.636296 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.636354 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 16:00:50.636511 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 16:00:50.636737 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12454,"mutex_wait_us":1,"thread_start_us":15304,"threads_started":1}
I20260929 16:00:50.638695 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.614564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.23.222.67:34437)
0929 16:00:50.630077 (+ 15513us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.630088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.630128 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:00:50.630254 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.630261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.630538 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.630757 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:50.631763 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.633096 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.635785 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.637192 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.637336 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.637559 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.638141 (+ 582us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.638332 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15324,"thread_start_us":15156,"threads_started":1}
I20260929 16:00:50.639158 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.614032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:34437 (local address 127.0.0.1:53112)
0929 16:00:50.628889 (+ 14857us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:50.628919 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:50.629114 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:50.631048 (+ 1934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:50.631053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:50.631079 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:50.631525 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.631539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.633291 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.633297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.635596 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.635612 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.637731 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.637737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.637915 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.637946 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 16:00:50.638035 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 16:00:50.638164 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14659,"thread_start_us":12156,"threads_started":1}
I20260929 16:00:50.640043 6394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.616649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.0.0.1:54868)
0929 16:00:50.630417 (+ 13768us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:50.630436 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:50.630684 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:50.631798 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:50.631804 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:50.631837 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:50.632370 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.632383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.634432 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.634438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.636559 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.636570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.639042 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.639049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.639378 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.639415 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:00:50.639529 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 16:00:50.639659 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13642,"mutex_wait_us":62,"thread_start_us":13455,"threads_started":1}
I20260929 16:00:50.640074 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.617323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.23.222.65:36375)
0929 16:00:50.624538 (+ 7215us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.624552 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.630564 (+ 6012us) server_negotiation.cc:407] Connection header received
0929 16:00:50.630910 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.630920 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.631154 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.631512 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:50.632581 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.634218 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.636748 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.638353 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.638496 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.638715 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.639577 (+ 862us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.639748 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6933,"thread_start_us":6714,"threads_started":1}
I20260929 16:00:50.641294 6177 tablet_service.cc:1467] Processing CreateTablet for tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b (DEFAULT_TABLE table=test-table [id=b9b3d9662566499e9e4bb0129dc7f42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:50.642753 6177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:50.643851 6042 tablet_service.cc:1467] Processing CreateTablet for tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b (DEFAULT_TABLE table=test-table [id=b9b3d9662566499e9e4bb0129dc7f42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:50.645334 6042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:50.644906 6312 tablet_service.cc:1467] Processing CreateTablet for tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b (DEFAULT_TABLE table=test-table [id=b9b3d9662566499e9e4bb0129dc7f42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:50.646301 6312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:50.655186 6396 tablet_bootstrap.cc:492] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: Bootstrap starting.
I20260929 16:00:50.658931 6396 tablet_bootstrap.cc:654] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:50.660094 6396 log.cc:824] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:50.662390 6397 tablet_bootstrap.cc:492] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: Bootstrap starting.
I20260929 16:00:50.663180 6396 tablet_bootstrap.cc:492] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: No bootstrap required, opened a new log
I20260929 16:00:50.663493 6396 ts_tablet_manager.cc:1397] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:00:50.665668 6397 tablet_bootstrap.cc:654] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:50.666838 6397 log.cc:824] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:50.668107 6398 tablet_bootstrap.cc:492] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: Bootstrap starting.
I20260929 16:00:50.672883 6398 tablet_bootstrap.cc:654] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:50.672994 6397 tablet_bootstrap.cc:492] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: No bootstrap required, opened a new log
I20260929 16:00:50.673525 6397 ts_tablet_manager.cc:1397] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260929 16:00:50.674821 6398 log.cc:824] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:50.677206 6396 raft_consensus.cc:348] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } }
I20260929 16:00:50.677766 6396 raft_consensus.cc:374] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:50.677868 6396 raft_consensus.cc:729] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436d07338a374dfcb504c48028129c58, State: Initialized, Role: FOLLOWER
I20260929 16:00:50.678687 6396 consensus_queue.cc:260] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } }
I20260929 16:00:50.685268 6398 tablet_bootstrap.cc:492] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: No bootstrap required, opened a new log
I20260929 16:00:50.685672 6398 ts_tablet_manager.cc:1397] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20260929 16:00:50.686993 6396 ts_tablet_manager.cc:1428] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: Time spent starting tablet: real 0.023s user 0.015s sys 0.004s
I20260929 16:00:50.687423 6243 heartbeater.cc:503] Master 127.23.222.126:42523 was elected leader, sending a full tablet report...
I20260929 16:00:50.688521 6397 raft_consensus.cc:348] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } }
I20260929 16:00:50.689054 6397 raft_consensus.cc:374] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:50.689177 6397 raft_consensus.cc:729] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f6ea2be6cb4dc499590ce2b5c8a896, State: Initialized, Role: FOLLOWER
I20260929 16:00:50.689987 6397 consensus_queue.cc:260] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } }
I20260929 16:00:50.692051 6377 heartbeater.cc:503] Master 127.23.222.126:42523 was elected leader, sending a full tablet report...
I20260929 16:00:50.692842 6397 ts_tablet_manager.cc:1428] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: Time spent starting tablet: real 0.019s user 0.019s sys 0.001s
I20260929 16:00:50.700416 6398 raft_consensus.cc:348] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } }
I20260929 16:00:50.701001 6398 raft_consensus.cc:374] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:50.701121 6398 raft_consensus.cc:729] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c22709258bbf47e9b5c50ddf62c3702c, State: Initialized, Role: FOLLOWER
I20260929 16:00:50.701778 6398 consensus_queue.cc:260] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } }
I20260929 16:00:50.703918 6398 ts_tablet_manager.cc:1428] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
W20260929 16:00:50.723608 6109 tablet.cc:2369] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:00:50.765556 6244 tablet.cc:2369] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:00:50.778848 6378 tablet.cc:2369] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:00:50.790117 6402 raft_consensus.cc:482] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:00:50.790408 6402 raft_consensus.cc:504] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } }
I20260929 16:00:50.792250 6402 leader_election.cc:288] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437), c22709258bbf47e9b5c50ddf62c3702c (127.23.222.65:36375)
I20260929 16:00:50.799502 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.792751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:51711 (local address 127.23.222.65:36375)
0929 16:00:50.792982 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.792988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.793866 (+ 878us) server_negotiation.cc:407] Connection header received
0929 16:00:50.794069 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.794077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.794167 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.794237 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:50.794902 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.795721 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.797654 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.798471 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.798635 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.798728 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.799054 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.799200 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:00:50.799533 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.792812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:39011 (local address 127.23.222.67:34437)
0929 16:00:50.793022 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.793030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.793949 (+ 919us) server_negotiation.cc:407] Connection header received
0929 16:00:50.794138 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.794142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.794202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.794330 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:50.795051 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.795952 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.797731 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.798484 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.798691 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.798766 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.799111 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.799276 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:00:50.799692 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.792940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:34437 (local address 127.23.222.66:39011)
0929 16:00:50.793801 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:50.793845 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:50.793999 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:50.794359 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:50.794363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:50.794390 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:50.794890 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.794902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.796088 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.796091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.797580 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.797597 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.798750 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.798757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.798891 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.798955 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 16:00:50.799066 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:00:50.799193 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":76,"thread_start_us":515,"threads_started":1}
I20260929 16:00:50.800189 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.792852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.23.222.66:51711)
0929 16:00:50.793746 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:50.793762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:50.793890 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:50.794303 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:50.794311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:50.794337 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:50.794755 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.794764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.795856 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.795861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.797493 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:50.797505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.798719 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.798729 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:50.798879 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.798924 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:00:50.798997 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 16:00:50.799154 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":78,"thread_start_us":491,"threads_started":1}
I20260929 16:00:50.800808 6332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fd808bdd0e427ca1cbe9f10ad2ea0b" candidate_uuid: "436d07338a374dfcb504c48028129c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" is_pre_election: true
I20260929 16:00:50.801201 6062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fd808bdd0e427ca1cbe9f10ad2ea0b" candidate_uuid: "436d07338a374dfcb504c48028129c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22709258bbf47e9b5c50ddf62c3702c" is_pre_election: true
I20260929 16:00:50.801441 6332 raft_consensus.cc:2454] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436d07338a374dfcb504c48028129c58 in term 0.
I20260929 16:00:50.801762 6062 raft_consensus.cc:2454] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436d07338a374dfcb504c48028129c58 in term 0.
I20260929 16:00:50.802206 6133 leader_election.cc:302] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f6ea2be6cb4dc499590ce2b5c8a896, 436d07338a374dfcb504c48028129c58; no voters:
I20260929 16:00:50.802588 6402 raft_consensus.cc:2781] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:00:50.802774 6402 raft_consensus.cc:482] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:00:50.802888 6402 raft_consensus.cc:3037] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:50.805579 6402 raft_consensus.cc:504] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } }
I20260929 16:00:50.806201 6402 leader_election.cc:288] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [CANDIDATE]: Term 1 election: Requested vote from peers 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437), c22709258bbf47e9b5c50ddf62c3702c (127.23.222.65:36375)
I20260929 16:00:50.806746 6332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fd808bdd0e427ca1cbe9f10ad2ea0b" candidate_uuid: "436d07338a374dfcb504c48028129c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896"
I20260929 16:00:50.806746 6062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30fd808bdd0e427ca1cbe9f10ad2ea0b" candidate_uuid: "436d07338a374dfcb504c48028129c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22709258bbf47e9b5c50ddf62c3702c"
I20260929 16:00:50.806970 6062 raft_consensus.cc:3037] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:50.806978 6332 raft_consensus.cc:3037] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:50.810279 6332 raft_consensus.cc:2454] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436d07338a374dfcb504c48028129c58 in term 1.
I20260929 16:00:50.810279 6062 raft_consensus.cc:2454] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436d07338a374dfcb504c48028129c58 in term 1.
I20260929 16:00:50.810703 6133 leader_election.cc:302] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f6ea2be6cb4dc499590ce2b5c8a896, 436d07338a374dfcb504c48028129c58; no voters:
I20260929 16:00:50.811060 6402 raft_consensus.cc:2781] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:00:50.811548 6402 raft_consensus.cc:686] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 1 LEADER]: Becoming Leader. State: Replica: 436d07338a374dfcb504c48028129c58, State: Running, Role: LEADER
I20260929 16:00:50.812055 6402 consensus_queue.cc:237] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } }
I20260929 16:00:50.816884 5914 catalog_manager.cc:5495] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 436d07338a374dfcb504c48028129c58 (127.23.222.66). New cstate: current_term: 1 leader_uuid: "436d07338a374dfcb504c48028129c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } health_report { overall_health: UNKNOWN } } }
I20260929 16:00:50.838888 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.833289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.23.222.66:42811)
0929 16:00:50.833502 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.833507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.833527 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:00:50.833785 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.833791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.833913 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.834100 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:00:50.834422 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.835190 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.836860 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.837274 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.837378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.837844 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:00:50.838358 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:00:50.838468 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.838593 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.838662 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:00:50.847357 6062 raft_consensus.cc:1261] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Refusing update from remote peer 436d07338a374dfcb504c48028129c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:50.847347 6332 raft_consensus.cc:1261] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 1 FOLLOWER]: Refusing update from remote peer 436d07338a374dfcb504c48028129c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:50.848371 6407 consensus_queue.cc:1035] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:00:50.848775 6402 consensus_queue.cc:1035] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:00:50.861312 6410 mvcc.cc:205] Tried to move back new op lower bound from 7334697577860096000 to 7334697577730076672. Current Snapshot: MvccSnapshot[applied={T|T < 7334697577860096000}]
I20260929 16:00:50.875710 5914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:00:50.876032 5914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35352:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:00:50.880035 5914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Sending DeleteTablet for 3 replicas of tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b
I20260929 16:00:50.881255 6177 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:50 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:52340
I20260929 16:00:50.881299 6312 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:50 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:53112
I20260929 16:00:50.882017 6042 tablet_service.cc:1514] Processing DeleteTablet for tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:50 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:54868
I20260929 16:00:50.882094 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 5978
I20260929 16:00:50.883267 6422 tablet_replica.cc:330] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: stopping tablet replica
I20260929 16:00:50.883867 6422 raft_consensus.cc:2229] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:00:50.884526 6422 raft_consensus.cc:2258] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:50.885512 6421 tablet_replica.cc:330] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: stopping tablet replica
I20260929 16:00:50.886065 6421 raft_consensus.cc:2229] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:00:50.886865 6421 raft_consensus.cc:2258] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:50.890048 6422 ts_tablet_manager.cc:1905] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:50.890038 6421 ts_tablet_manager.cc:1905] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:50.899814 6420 tablet_replica.cc:330] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: stopping tablet replica
I20260929 16:00:50.900291 6420 raft_consensus.cc:2229] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:00:50.900787 6420 raft_consensus.cc:2258] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:50.902273 6421 ts_tablet_manager.cc:1918] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:50.902505 6421 log.cc:1196] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/30fd808bdd0e427ca1cbe9f10ad2ea0b
I20260929 16:00:50.903193 6421 ts_tablet_manager.cc:1939] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 436d07338a374dfcb504c48028129c58: Deleting consensus metadata
I20260929 16:00:50.903311 6420 ts_tablet_manager.cc:1905] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:50.905535 5901 catalog_manager.cc:4841] TS 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811): tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b (table test-table [id=b9b3d9662566499e9e4bb0129dc7f42b]) successfully deleted
I20260929 16:00:50.907701 6422 ts_tablet_manager.cc:1918] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:50.907904 6422 log.cc:1196] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/30fd808bdd0e427ca1cbe9f10ad2ea0b
I20260929 16:00:50.908486 6422 ts_tablet_manager.cc:1939] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P c22709258bbf47e9b5c50ddf62c3702c: Deleting consensus metadata
I20260929 16:00:50.910573 5900 catalog_manager.cc:4841] TS c22709258bbf47e9b5c50ddf62c3702c (127.23.222.65:36375): tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b (table test-table [id=b9b3d9662566499e9e4bb0129dc7f42b]) successfully deleted
I20260929 16:00:50.911429 6420 ts_tablet_manager.cc:1918] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:50.911675 6420 log.cc:1196] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/30fd808bdd0e427ca1cbe9f10ad2ea0b
I20260929 16:00:50.912182 6420 ts_tablet_manager.cc:1939] T 30fd808bdd0e427ca1cbe9f10ad2ea0b P 19f6ea2be6cb4dc499590ce2b5c8a896: Deleting consensus metadata
I20260929 16:00:50.913769 5902 catalog_manager.cc:4841] TS 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437): tablet 30fd808bdd0e427ca1cbe9f10ad2ea0b (table test-table [id=b9b3d9662566499e9e4bb0129dc7f42b]) successfully deleted
Sep 29 16:00:50 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 16:00:50.922768 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:50.883517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.23.222.65:36375)
0929 16:00:50.883693 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:50.883710 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:50.903333 (+ 19623us) server_negotiation.cc:407] Connection header received
0929 16:00:50.903659 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:50.903665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:50.903798 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:50.903882 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:50.906287 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.907511 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.909637 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:50.910329 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:50.910437 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:50.915938 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:50.915957 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:50.915977 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:50.916025 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:50.919856 (+ 3831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:50.920597 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:50.920607 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:50.920610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:50.920782 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:50.921137 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:50.921143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:50.921146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:50.921619 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:50.921953 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:50.922141 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 16:00:50.922373 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:00:50.923683 6103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:51.056566 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 5978
I20260929 16:00:51.075434 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 6114
Sep 29 16:00:51 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 16:00:51.098366 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:51.076300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.23.222.66:42811)
0929 16:00:51.076866 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:51.076873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:51.076892 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:00:51.076957 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:51.076963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:51.077063 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:51.077175 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:51.078516 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:51.079864 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:51.081536 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:51.082258 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:51.082400 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:51.091037 (+ 8637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:51.091059 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:51.091078 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:51.091132 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:51.095106 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:51.096091 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:51.096103 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:51.096108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:51.096281 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:51.096630 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:51.096635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:51.096640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:51.097119 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:51.097451 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:51.097756 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 16:00:51.097990 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429}
I20260929 16:00:51.099263 6237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:51.217657 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 6114
I20260929 16:00:51.236999 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 6249
Sep 29 16:00:51 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 16:00:51.253849 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:51.237908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.23.222.67:34437)
0929 16:00:51.238135 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:51.238149 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:51.238251 (+ 102us) server_negotiation.cc:407] Connection header received
0929 16:00:51.238311 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:51.238315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:51.238385 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:51.238469 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:51.239814 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:51.240540 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:51.241858 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:51.242272 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:51.242377 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:51.247170 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:51.247186 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:51.247202 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:51.247306 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:51.250208 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:51.251035 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:51.251053 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:51.251060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:51.251365 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:51.251819 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:51.251823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:51.251825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:51.252277 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:51.252516 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:51.253241 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 16:00:51.253509 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:00:51.254577 6372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:51.345191 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 6249
I20260929 16:00:51.363188 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 5883
I20260929 16:00:51.364615 5946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:51.479218 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 5883
I20260929 16:00:51.501695 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42523
--webserver_interface=127.23.222.126
--webserver_port=43369
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:51.844008 6428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:51.844309 6428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:51.844475 6428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:51.851958 6428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:51.852083 6428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:51.852175 6428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:51.852285 6428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:51.852367 6428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:51.862746 6428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:42523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:51.864264 6428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:51.865661 6428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:51.889688 6437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:51.889655 6435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:51.889722 6434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:51.890470 6428 server_base.cc:1034] running on GCE node
I20260929 16:00:51.891520 6428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:51.893257 6428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:51.894574 6428 hybrid_clock.cc:648] HybridClock initialized: now 1790697651894507 us; error 63 us; skew 500 ppm
I20260929 16:00:51.895038 6428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:51 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697651, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:51.898756 6428 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:00:51.900525 6428 webserver.cc:460] Webserver started at http://127.23.222.126:43369/ using document root <none> and password file <none>
I20260929 16:00:51.901281 6428 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:51.901403 6428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:51.907984 6428 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:00:51.911342 6444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:51.912889 6428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:00:51.913051 6428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "29e858a095104ac4af470dba76519e8f"
format_stamp: "Formatted at 2026-09-29 16:00:48 on dist-test-slave-cq88"
server_key: "3d8e79b2f2efa33da8baa735e9320e3f"
server_key_iv: "cc886e18dfced2f30121963fcc4bba78"
server_key_version: "encryptionkey@0"
I20260929 16:00:51.914152 6428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:51.927553 6428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:51.932206 6428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:51.932459 6428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:51.952840 6428 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:42523
I20260929 16:00:51.952850 6495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:42523 every 8 connection(s)
I20260929 16:00:51.954602 6428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:00:51.955396 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 6428
I20260929 16:00:51.956425 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:36375
--local_ip_for_outbound_sockets=127.23.222.65
--tserver_master_addrs=127.23.222.126:42523
--webserver_port=34547
--webserver_interface=127.23.222.65
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:51.962505 6496 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:00:51.968098 6496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Bootstrap starting.
I20260929 16:00:51.998487 6496 log.cc:824] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:52.010847 6496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:00:52.011478 6496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Bootstrap complete.
I20260929 16:00:52.021481 6496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:52.022404 6496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e858a095104ac4af470dba76519e8f, State: Initialized, Role: FOLLOWER
I20260929 16:00:52.023113 6496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:52.023418 6496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:00:52.023540 6496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:00:52.023751 6496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:00:52.027817 6496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:52.028393 6496 leader_election.cc:302] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29e858a095104ac4af470dba76519e8f; no voters:
I20260929 16:00:52.029078 6496 leader_election.cc:288] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:00:52.029232 6500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:00:52.030550 6500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 2 LEADER]: Becoming Leader. State: Replica: 29e858a095104ac4af470dba76519e8f, State: Running, Role: LEADER
I20260929 16:00:52.031142 6500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:52.031675 6496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:52.033780 6502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29e858a095104ac4af470dba76519e8f. Latest consensus state: current_term: 2 leader_uuid: "29e858a095104ac4af470dba76519e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } } }
I20260929 16:00:52.034199 6502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:52.033957 6501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "29e858a095104ac4af470dba76519e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } } }
I20260929 16:00:52.034404 6501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:52.044045 6507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:00:52.053095 6507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:00:52.054852 6507 catalog_manager.cc:1259] Loaded cluster ID: 0f39dd2378e246c5a46967de256d16bf
I20260929 16:00:52.055089 6507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:00:52.058699 6507 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 16:00:52.076025 6507 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 16:00:52.085909 6507 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:00:52.087735 6507 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Loaded TSK: 0
I20260929 16:00:52.089504 6507 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:00:52.330313 6498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:52.330627 6498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:52.330781 6498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:52.338649 6498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:52.338783 6498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:52.339046 6498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:00:52.349731 6498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:36375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=34547
--tserver_master_addrs=127.23.222.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:52.351084 6498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:52.352545 6498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:52.367893 6523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:52.367811 6524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:52.368702 6527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:52.369290 6498 server_base.cc:1034] running on GCE node
I20260929 16:00:52.369952 6498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:52.370971 6498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:52.372267 6498 hybrid_clock.cc:648] HybridClock initialized: now 1790697652372189 us; error 57 us; skew 500 ppm
I20260929 16:00:52.372658 6498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:52 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:52.376183 6498 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:00:52.377873 6498 webserver.cc:460] Webserver started at http://127.23.222.65:34547/ using document root <none> and password file <none>
I20260929 16:00:52.378594 6498 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:52.378736 6498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:52.384508 6498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:00:52.387502 6534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:52.388964 6498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:00:52.389147 6498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "c22709258bbf47e9b5c50ddf62c3702c"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "842789e5f226299a63c44eb1176fef07"
server_key_iv: "3d2260844205cb813e3d6a6c0ccbbca7"
server_key_version: "encryptionkey@0"
I20260929 16:00:52.390167 6498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:52.430418 6498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:52.434911 6498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:52.435196 6498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:52.436357 6498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:52.437750 6498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:52.437892 6498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:52.438033 6498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:52.438127 6498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:52.472123 6498 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:36375
I20260929 16:00:52.472226 6646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:36375 every 8 connection(s)
I20260929 16:00:52.473687 6498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:00:52.474402 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 6498
I20260929 16:00:52.475307 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:42811
--local_ip_for_outbound_sockets=127.23.222.66
--tserver_master_addrs=127.23.222.126:42523
--webserver_port=43019
--webserver_interface=127.23.222.66
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:52 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:52.499621 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:52.476964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:46261 (local address 127.23.222.126:42523)
0929 16:00:52.477660 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:52.477678 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:52.478295 (+ 617us) server_negotiation.cc:407] Connection header received
0929 16:00:52.480558 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:52.480580 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:52.481043 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:52.481455 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:52.483469 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:52.484734 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:52.486321 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:52.486993 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:52.487197 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:52.491049 (+ 3852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:52.491062 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:52.491086 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:52.491152 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:52.494614 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:52.495828 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:52.495835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:52.495841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:52.496024 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:52.496584 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:52.496588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:52.496590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:52.497095 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:52.497244 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:52.498258 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0929 16:00:52.498565 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":287,"threads_started":1}
I20260929 16:00:52.500227 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:52.477111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:46261)
0929 16:00:52.478091 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:52.478145 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:52.479965 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:52.481561 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:52.481568 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:52.482350 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:52.483163 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:52.483193 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:52.484912 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:52.484918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:52.486158 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:52.486171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:52.487285 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:52.487294 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:52.487569 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:52.488318 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:52.488343 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:52.490716 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:52.494781 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:52.494807 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:52.494828 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:52.495623 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:52.496159 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:52.496167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:52.496172 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:52.496450 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:52.497246 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:52.497257 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:52.497737 (+ 480us) client_negotiation.cc:769] Sending connection context
0929 16:00:52.498082 (+ 345us) client_negotiation.cc:240] Negotiation successful
0929 16:00:52.498732 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":442,"threads_started":1}
I20260929 16:00:52.503353 6647 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42523
I20260929 16:00:52.503674 6647 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:52.504436 6647 heartbeater.cc:511] Master 127.23.222.126:42523 requested a full tablet report, sending...
I20260929 16:00:52.506910 6461 ts_manager.cc:194] Registered new tserver with Master: c22709258bbf47e9b5c50ddf62c3702c (127.23.222.65:36375)
I20260929 16:00:52.509086 6461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:46261
W20260929 16:00:52.820917 6651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:52.821197 6651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:52.821293 6651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:52.829883 6651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:52.830001 6651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:52.830120 6651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:00:52.840102 6651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:42811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=43019
--tserver_master_addrs=127.23.222.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:52.841355 6651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:52.842620 6651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:52.856643 6656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:52.856983 6659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:52.857637 6657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:52.857988 6651 server_base.cc:1034] running on GCE node
I20260929 16:00:52.858794 6651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:52.859884 6651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:52.861115 6651 hybrid_clock.cc:648] HybridClock initialized: now 1790697652861070 us; error 46 us; skew 500 ppm
I20260929 16:00:52.861456 6651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:52 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:52.864754 6651 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:00:52.866183 6651 webserver.cc:460] Webserver started at http://127.23.222.66:43019/ using document root <none> and password file <none>
I20260929 16:00:52.866876 6651 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:52.867039 6651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:52.872538 6651 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:00:52.875546 6666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:52.877074 6651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:00:52.877260 6651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "436d07338a374dfcb504c48028129c58"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "844b7a2772122ea52c4e4302236b21e0"
server_key_iv: "e12d93b0c3ccf06b39e976ad378879e9"
server_key_version: "encryptionkey@0"
I20260929 16:00:52.878271 6651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:52.893692 6651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:52.898307 6651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:52.898597 6651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:52.899645 6651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:52.901067 6651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:52.901185 6651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:52.901319 6651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:52.901407 6651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:52.934413 6651 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:42811
I20260929 16:00:52.934463 6780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:42811 every 8 connection(s)
I20260929 16:00:52.935951 6651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:00:52.941048 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 6651
I20260929 16:00:52.941654 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:34437
--local_ip_for_outbound_sockets=127.23.222.67
--tserver_master_addrs=127.23.222.126:42523
--webserver_port=36071
--webserver_interface=127.23.222.67
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:00:52 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697652, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:52.958698 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:52.938568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:58505 (local address 127.23.222.126:42523)
0929 16:00:52.938737 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:52.938741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:52.939884 (+ 1143us) server_negotiation.cc:407] Connection header received
0929 16:00:52.941756 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:52.941762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:52.941859 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:52.941971 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:52.943561 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:52.944272 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:52.946000 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:52.946792 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:52.946945 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:52.951346 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:52.951351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:52.951354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:52.951387 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:52.954787 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:52.955858 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:52.955863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:52.955867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:52.956050 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:52.956599 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:52.956605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:52.956608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:52.956982 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:52.957129 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:52.958082 (+ 953us) server_negotiation.cc:299] Negotiation successful
0929 16:00:52.958343 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:00:52.960194 6783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:52.938983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:58505)
0929 16:00:52.939722 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:52.939768 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:52.941485 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:52.942224 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:52.942232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:52.942727 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:52.943391 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:52.943412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:52.944435 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:52.944442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:52.945806 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:52.945819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:52.947019 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:52.947024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:52.947301 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:52.948059 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:52.948094 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:52.951075 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:52.954960 (+ 3885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:52.954972 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:52.954992 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:52.955693 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:52.956189 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:52.956197 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:52.956204 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:52.956457 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:52.957132 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:52.957143 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:52.957628 (+ 485us) client_negotiation.cc:769] Sending connection context
0929 16:00:52.957953 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 16:00:52.958565 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":322,"threads_started":1}
I20260929 16:00:52.962198 6781 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42523
I20260929 16:00:52.962571 6781 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:52.963629 6781 heartbeater.cc:511] Master 127.23.222.126:42523 requested a full tablet report, sending...
I20260929 16:00:52.965516 6461 ts_manager.cc:194] Registered new tserver with Master: 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811)
I20260929 16:00:52.966670 6461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:58505
W20260929 16:00:53.276321 6784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:53.276690 6784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:53.276868 6784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:53.284409 6784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:53.284590 6784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:53.284798 6784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:00:53.295034 6784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:34437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=36071
--tserver_master_addrs=127.23.222.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:53.296486 6784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:53.297899 6784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:53.311481 6790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:53.311708 6792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:53.311542 6789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:53.312580 6784 server_base.cc:1034] running on GCE node
I20260929 16:00:53.313192 6784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:53.314109 6784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:53.315376 6784 hybrid_clock.cc:648] HybridClock initialized: now 1790697653315319 us; error 40 us; skew 500 ppm
I20260929 16:00:53.315768 6784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:53 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697653, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:53.319137 6784 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:00:53.320919 6784 webserver.cc:460] Webserver started at http://127.23.222.67:36071/ using document root <none> and password file <none>
I20260929 16:00:53.321651 6784 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:53.321777 6784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:53.327721 6784 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:00:53.330704 6799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:53.332369 6784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:00:53.332520 6784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "19f6ea2be6cb4dc499590ce2b5c8a896"
format_stamp: "Formatted at 2026-09-29 16:00:50 on dist-test-slave-cq88"
server_key: "257d5901640b32873a6b4eda331c80cc"
server_key_iv: "4404c6b7265617ecdc8f6a15dbdb5816"
server_key_version: "encryptionkey@0"
I20260929 16:00:53.333634 6784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:53.353242 6784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:53.358268 6784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:53.358526 6784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:53.359704 6784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:53.361217 6784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:53.361313 6784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:53.361395 6784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:53.361460 6784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:53.398636 6784 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:34437
I20260929 16:00:53.398682 6912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:34437 every 8 connection(s)
I20260929 16:00:53.400300 6784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:00:53.407829 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 6784
Sep 29 16:00:53 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697653, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:00:53.423029 6649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.403082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:34493 (local address 127.23.222.126:42523)
0929 16:00:53.403338 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.403346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.404476 (+ 1130us) server_negotiation.cc:407] Connection header received
0929 16:00:53.406326 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.406331 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.406410 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.406529 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:53.408210 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.409251 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.411434 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.412111 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.412216 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.416456 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:53.416461 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:53.416464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:53.416496 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:53.419958 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.420698 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.420703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.420706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.420841 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.421294 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.421306 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.421308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.421577 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:53.421750 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.422451 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.422693 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:00:53.424327 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.403540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:34493)
0929 16:00:53.404295 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.404340 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:53.406017 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:53.406739 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:53.406747 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:53.407364 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:53.407992 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.408020 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.409494 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.409501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.411222 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.411287 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.412364 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.412373 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.412708 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.413538 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:53.413565 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:53.416187 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:53.420106 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:53.420114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:53.420132 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:53.420569 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:53.420964 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:53.420969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:53.420973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:53.421168 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:53.421710 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:53.421716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:53.422057 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 16:00:53.422350 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 16:00:53.422868 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":292,"threads_started":1}
I20260929 16:00:53.424402 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.408916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.23.222.126:42523)
0929 16:00:53.409489 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.409496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.409963 (+ 467us) server_negotiation.cc:407] Connection header received
0929 16:00:53.410093 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.410099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.410180 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.410306 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:53.412024 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.413243 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.414654 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.415156 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.415357 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.418197 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:53.418205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:53.418212 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:53.418268 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:53.421544 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.422263 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.422269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.422274 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.422452 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.422869 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.422874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.422876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.423212 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:53.423484 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.423789 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.424005 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":452,"spinlock_wait_cycles":3456,"thread_start_us":278,"threads_started":1}
I20260929 16:00:53.425925 6913 heartbeater.cc:347] Connected to a master server at 127.23.222.126:42523
I20260929 16:00:53.426288 6913 heartbeater.cc:464] Registering TS with master...
I20260929 16:00:53.427284 6913 heartbeater.cc:511] Master 127.23.222.126:42523 requested a full tablet report, sending...
I20260929 16:00:53.428720 6461 ts_manager.cc:194] Registered new tserver with Master: 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437)
I20260929 16:00:53.429491 6461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:34493
I20260929 16:00:53.440037 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:00:53.447292 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.440987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.23.222.126:42523)
0929 16:00:53.441162 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.441167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.441693 (+ 526us) server_negotiation.cc:407] Connection header received
0929 16:00:53.441871 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.441875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.442026 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.442152 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:00:53.442624 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.443410 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.445088 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.445596 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.445695 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.446180 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:00:53.446734 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:00:53.446864 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.446974 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.447041 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:00:53.448297 6461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:00:53.455708 6461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:00:53.489197 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.468505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:34437 (local address 127.0.0.1:49442)
0929 16:00:53.480669 (+ 12164us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.480685 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:53.480800 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:53.481672 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:53.481680 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:53.481711 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:53.482207 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.482220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.484445 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.484451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.486602 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.486614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.488213 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.488220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.488417 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.488484 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 16:00:53.488631 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 16:00:53.488835 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12036,"thread_start_us":7671,"threads_started":1}
I20260929 16:00:53.489794 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.469575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.23.222.67:34437)
0929 16:00:53.478353 (+ 8778us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.478364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.480911 (+ 2547us) server_negotiation.cc:407] Connection header received
0929 16:00:53.481001 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.481006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.481205 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.481361 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:53.482452 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.484273 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.486762 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.487987 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.488138 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.488391 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.488672 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.488837 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8549,"thread_start_us":12109,"threads_started":1}
I20260929 16:00:53.491673 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.468083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:42811 (local address 127.0.0.1:60580)
0929 16:00:53.480361 (+ 12278us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.480387 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:53.480551 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:53.482800 (+ 2249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:53.482804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:53.482828 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:53.483178 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.483190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.486130 (+ 2940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.486135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.488332 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.488347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.490786 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.490792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.490977 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.491011 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:00:53.491122 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 16:00:53.491276 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12120,"mutex_wait_us":2,"thread_start_us":11981,"threads_started":1}
I20260929 16:00:53.491732 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.469190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.23.222.66:42811)
0929 16:00:53.477765 (+ 8575us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.477780 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.480695 (+ 2915us) server_negotiation.cc:407] Connection header received
0929 16:00:53.480812 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.480822 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.481092 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.481305 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:53.483401 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.485988 (+ 2587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.489010 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.490522 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.490662 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.490846 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.491211 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.491421 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8355,"thread_start_us":8227,"threads_started":1}
I20260929 16:00:53.493767 6847 tablet_service.cc:1467] Processing CreateTablet for tablet ad399c6e5c054311a3ee83a63bcb4a82 (DEFAULT_TABLE table=test-table [id=59157dba92994561a0ac23d60bb4519f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:53.494509 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.467787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.0.0.1:59174)
0929 16:00:53.476741 (+ 8954us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.476806 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:53.477052 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:53.486836 (+ 9784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:53.486841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:53.486866 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:53.487333 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.487349 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.489179 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.489185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.491436 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.491450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.493723 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.493731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.493929 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.493963 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:00:53.494054 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 16:00:53.494176 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8594,"thread_start_us":13694,"threads_started":1}
I20260929 16:00:53.494632 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.468031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.23.222.65:36375)
0929 16:00:53.485799 (+ 17768us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.485807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.486217 (+ 410us) server_negotiation.cc:407] Connection header received
0929 16:00:53.486335 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.486344 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.486600 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.486822 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:53.487531 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.489016 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.491624 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.493147 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.493291 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.493503 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.494148 (+ 645us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.494303 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17576,"thread_start_us":17435,"threads_started":1}
I20260929 16:00:53.495579 6847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad399c6e5c054311a3ee83a63bcb4a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:53.495589 6715 tablet_service.cc:1467] Processing CreateTablet for tablet ad399c6e5c054311a3ee83a63bcb4a82 (DEFAULT_TABLE table=test-table [id=59157dba92994561a0ac23d60bb4519f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:53.496924 6715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad399c6e5c054311a3ee83a63bcb4a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:53.498159 6582 tablet_service.cc:1467] Processing CreateTablet for tablet ad399c6e5c054311a3ee83a63bcb4a82 (DEFAULT_TABLE table=test-table [id=59157dba92994561a0ac23d60bb4519f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:00:53.499475 6582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad399c6e5c054311a3ee83a63bcb4a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:00:53.507941 6929 tablet_bootstrap.cc:492] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: Bootstrap starting.
I20260929 16:00:53.511590 6929 tablet_bootstrap.cc:654] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:53.512449 6647 heartbeater.cc:503] Master 127.23.222.126:42523 was elected leader, sending a full tablet report...
I20260929 16:00:53.513042 6929 log.cc:824] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:53.515724 6932 tablet_bootstrap.cc:492] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Bootstrap starting.
I20260929 16:00:53.517359 6930 tablet_bootstrap.cc:492] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: Bootstrap starting.
I20260929 16:00:53.520514 6932 tablet_bootstrap.cc:654] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:53.520800 6930 tablet_bootstrap.cc:654] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: Neither blocks nor log segments found. Creating new log.
I20260929 16:00:53.521847 6930 log.cc:824] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:53.522042 6932 log.cc:824] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:53.524886 6932 tablet_bootstrap.cc:492] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: No bootstrap required, opened a new log
I20260929 16:00:53.525135 6930 tablet_bootstrap.cc:492] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: No bootstrap required, opened a new log
I20260929 16:00:53.525138 6929 tablet_bootstrap.cc:492] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: No bootstrap required, opened a new log
I20260929 16:00:53.525164 6932 ts_tablet_manager.cc:1397] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 16:00:53.525401 6929 ts_tablet_manager.cc:1397] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260929 16:00:53.525401 6930 ts_tablet_manager.cc:1397] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 16:00:53.532819 6932 raft_consensus.cc:348] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:53.533219 6932 raft_consensus.cc:374] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:53.533313 6932 raft_consensus.cc:729] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c22709258bbf47e9b5c50ddf62c3702c, State: Initialized, Role: FOLLOWER
I20260929 16:00:53.533960 6932 consensus_queue.cc:260] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:53.535925 6932 ts_tablet_manager.cc:1428] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260929 16:00:53.536327 6930 raft_consensus.cc:348] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:53.536325 6929 raft_consensus.cc:348] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:53.536800 6930 raft_consensus.cc:374] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:53.536794 6929 raft_consensus.cc:374] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:00:53.536903 6930 raft_consensus.cc:729] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436d07338a374dfcb504c48028129c58, State: Initialized, Role: FOLLOWER
I20260929 16:00:53.536903 6929 raft_consensus.cc:729] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19f6ea2be6cb4dc499590ce2b5c8a896, State: Initialized, Role: FOLLOWER
I20260929 16:00:53.537529 6929 consensus_queue.cc:260] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:53.537529 6930 consensus_queue.cc:260] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:53.539938 6781 heartbeater.cc:503] Master 127.23.222.126:42523 was elected leader, sending a full tablet report...
I20260929 16:00:53.540135 6913 heartbeater.cc:503] Master 127.23.222.126:42523 was elected leader, sending a full tablet report...
I20260929 16:00:53.540858 6929 ts_tablet_manager.cc:1428] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: Time spent starting tablet: real 0.015s user 0.010s sys 0.007s
I20260929 16:00:53.541764 6930 ts_tablet_manager.cc:1428] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260929 16:00:53.568990 6936 raft_consensus.cc:482] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:00:53.569314 6936 raft_consensus.cc:504] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:53.571054 6936 leader_election.cc:288] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c22709258bbf47e9b5c50ddf62c3702c (127.23.222.65:36375), 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437)
I20260929 16:00:53.579584 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.571670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:54761 (local address 127.23.222.67:34437)
0929 16:00:53.571923 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.571931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.572864 (+ 933us) server_negotiation.cc:407] Connection header received
0929 16:00:53.573075 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.573079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.573146 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.573247 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:53.573869 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.574818 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.577826 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.578620 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.578762 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.578818 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.579146 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.579306 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 16:00:53.579936 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.571633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:34437 (local address 127.23.222.66:54761)
0929 16:00:53.572731 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.572754 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:53.572905 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:53.573263 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:53.573268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:53.573290 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:53.573715 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.573726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.574962 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.574967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.577639 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.577652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.578828 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.578832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.578948 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.578974 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:00:53.579092 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:00:53.579233 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"mutex_wait_us":25,"thread_start_us":505,"threads_started":1}
I20260929 16:00:53.580991 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad399c6e5c054311a3ee83a63bcb4a82" candidate_uuid: "436d07338a374dfcb504c48028129c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" is_pre_election: true
I20260929 16:00:53.581195 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.571663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:40193 (local address 127.23.222.65:36375)
0929 16:00:53.571908 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.571916 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.572989 (+ 1073us) server_negotiation.cc:407] Connection header received
0929 16:00:53.573197 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.573201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.573263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.573365 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:00:53.574036 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.575096 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.578940 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.580157 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.580314 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.580392 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.580813 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.580948 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:00:53.581387 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.571941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.23.222.66:40193)
0929 16:00:53.572867 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.572886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:53.573060 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:53.573380 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:53.573384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:53.573423 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:00:53.573885 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.573895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.575250 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.575257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.577635 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:53.577652 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.580515 (+ 2863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.580520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:53.580648 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.580671 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:00:53.580784 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 16:00:53.580914 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":90,"thread_start_us":553,"threads_started":1}
I20260929 16:00:53.581557 6868 raft_consensus.cc:2454] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436d07338a374dfcb504c48028129c58 in term 0.
I20260929 16:00:53.582283 6670 leader_election.cc:302] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f6ea2be6cb4dc499590ce2b5c8a896, 436d07338a374dfcb504c48028129c58; no voters:
I20260929 16:00:53.582681 6936 raft_consensus.cc:2781] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:00:53.582549 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad399c6e5c054311a3ee83a63bcb4a82" candidate_uuid: "436d07338a374dfcb504c48028129c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22709258bbf47e9b5c50ddf62c3702c" is_pre_election: true
I20260929 16:00:53.582866 6936 raft_consensus.cc:482] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:00:53.582971 6936 raft_consensus.cc:3037] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:53.583135 6602 raft_consensus.cc:2454] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 436d07338a374dfcb504c48028129c58 in term 0.
I20260929 16:00:53.585481 6936 raft_consensus.cc:504] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:53.586164 6936 leader_election.cc:288] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [CANDIDATE]: Term 1 election: Requested vote from peers c22709258bbf47e9b5c50ddf62c3702c (127.23.222.65:36375), 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437)
I20260929 16:00:53.586612 6602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad399c6e5c054311a3ee83a63bcb4a82" candidate_uuid: "436d07338a374dfcb504c48028129c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22709258bbf47e9b5c50ddf62c3702c"
I20260929 16:00:53.586706 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad399c6e5c054311a3ee83a63bcb4a82" candidate_uuid: "436d07338a374dfcb504c48028129c58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896"
I20260929 16:00:53.586898 6602 raft_consensus.cc:3037] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:53.586926 6868 raft_consensus.cc:3037] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:00:53.590237 6868 raft_consensus.cc:2454] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436d07338a374dfcb504c48028129c58 in term 1.
I20260929 16:00:53.590230 6602 raft_consensus.cc:2454] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 436d07338a374dfcb504c48028129c58 in term 1.
I20260929 16:00:53.590685 6670 leader_election.cc:302] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19f6ea2be6cb4dc499590ce2b5c8a896, 436d07338a374dfcb504c48028129c58; no voters:
I20260929 16:00:53.591017 6936 raft_consensus.cc:2781] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:00:53.591543 6936 raft_consensus.cc:686] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 1 LEADER]: Becoming Leader. State: Replica: 436d07338a374dfcb504c48028129c58, State: Running, Role: LEADER
I20260929 16:00:53.592029 6936 consensus_queue.cc:237] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:53.596745 6459 catalog_manager.cc:5495] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 436d07338a374dfcb504c48028129c58 (127.23.222.66). New cstate: current_term: 1 leader_uuid: "436d07338a374dfcb504c48028129c58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } health_report { overall_health: UNKNOWN } } }
I20260929 16:00:53.616286 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.610666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.23.222.66:42811)
0929 16:00:53.610882 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.610888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.610991 (+ 103us) server_negotiation.cc:407] Connection header received
0929 16:00:53.611278 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.611283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.611373 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.611525 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:00:53.611933 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.612668 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.614007 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.614452 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.614590 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.615116 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:00:53.615675 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:00:53.615843 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.615909 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.615993 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:00:53.624847 6602 raft_consensus.cc:1261] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Refusing update from remote peer 436d07338a374dfcb504c48028129c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:53.624846 6868 raft_consensus.cc:1261] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 1 FOLLOWER]: Refusing update from remote peer 436d07338a374dfcb504c48028129c58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:00:53.625851 6940 consensus_queue.cc:1035] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:00:53.626207 6936 consensus_queue.cc:1035] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:00:53.638203 6942 mvcc.cc:205] Tried to move back new op lower bound from 7334697589237334016 to 7334697589117136896. Current Snapshot: MvccSnapshot[applied={T|T < 7334697589237334016}]
I20260929 16:00:53.653391 6460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:00:53.653647 6460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:00:53.657248 6460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Sending DeleteTablet for 3 replicas of tablet ad399c6e5c054311a3ee83a63bcb4a82
I20260929 16:00:53.658499 6582 tablet_service.cc:1514] Processing DeleteTablet for tablet ad399c6e5c054311a3ee83a63bcb4a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:53 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:59174
I20260929 16:00:53.658556 6715 tablet_service.cc:1514] Processing DeleteTablet for tablet ad399c6e5c054311a3ee83a63bcb4a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:53 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:60580
I20260929 16:00:53.658785 6847 tablet_service.cc:1514] Processing DeleteTablet for tablet ad399c6e5c054311a3ee83a63bcb4a82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:00:53 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:49442
I20260929 16:00:53.660028 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 6498
I20260929 16:00:53.670097 6954 tablet_replica.cc:330] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: stopping tablet replica
I20260929 16:00:53.670717 6954 raft_consensus.cc:2229] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:00:53.671957 6953 tablet_replica.cc:330] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: stopping tablet replica
I20260929 16:00:53.672369 6954 raft_consensus.cc:2258] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:53.673070 6953 raft_consensus.cc:2229] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:00:53.674029 6953 raft_consensus.cc:2258] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:00:53.676218 6954 ts_tablet_manager.cc:1905] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:53.676326 6953 ts_tablet_manager.cc:1905] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:00:53.680374 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.662317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.23.222.65:36375)
0929 16:00:53.662540 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.662550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.662574 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:00:53.662650 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.662658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.662756 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.662881 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:53.664604 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.666117 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.668047 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.668915 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.669058 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.673019 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:53.673041 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:53.673064 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:53.673111 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:53.677128 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.678002 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.678012 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.678016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.678171 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.678553 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.678557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.678560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.679072 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:53.679423 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.679736 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.679976 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:00:53.681142 6642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:53.685803 6953 ts_tablet_manager.cc:1918] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:53.686049 6953 log.cc:1196] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/ad399c6e5c054311a3ee83a63bcb4a82
I20260929 16:00:53.686806 6953 ts_tablet_manager.cc:1939] T ad399c6e5c054311a3ee83a63bcb4a82 P 436d07338a374dfcb504c48028129c58: Deleting consensus metadata
I20260929 16:00:53.687448 6954 ts_tablet_manager.cc:1918] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:00:53.687705 6954 log.cc:1196] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/ad399c6e5c054311a3ee83a63bcb4a82
I20260929 16:00:53.688536 6954 ts_tablet_manager.cc:1939] T ad399c6e5c054311a3ee83a63bcb4a82 P 19f6ea2be6cb4dc499590ce2b5c8a896: Deleting consensus metadata
I20260929 16:00:53.689774 6447 catalog_manager.cc:4841] TS 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811): tablet ad399c6e5c054311a3ee83a63bcb4a82 (table test-table [id=59157dba92994561a0ac23d60bb4519f]) successfully deleted
I20260929 16:00:53.691538 6448 catalog_manager.cc:4841] TS 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437): tablet ad399c6e5c054311a3ee83a63bcb4a82 (table test-table [id=59157dba92994561a0ac23d60bb4519f]) successfully deleted
I20260929 16:00:53.780759 6952 tablet_replica.cc:330] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: stopping tablet replica
I20260929 16:00:53.780964 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 6498
W20260929 16:00:53.799837 6446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.222.65:36375 (error 108)
I20260929 16:00:53.800047 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 6651
I20260929 16:00:53.801996 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.801490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.0.0.1:59200)
0929 16:00:53.801686 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.801819 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:36375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 16:00:53.802289 6446 catalog_manager.cc:4584] TS c22709258bbf47e9b5c50ddf62c3702c (127.23.222.65:36375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad399c6e5c054311a3ee83a63bcb4a82: Network error: Client connection negotiation failed: client connection to 127.23.222.65:36375: connect: Connection refused (error 111)
I20260929 16:00:53.824077 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.800990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.23.222.66:42811)
0929 16:00:53.801203 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.801207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.801225 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:00:53.801373 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.801378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.801501 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.801630 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:53.804128 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.805084 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.807056 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.807716 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.807849 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.817230 (+ 9381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:53.817249 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:53.817272 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:53.817333 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:53.820692 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.821707 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.821721 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.821726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.821934 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.822423 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.822427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.822429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.822855 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:53.823115 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.823539 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.823720 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:00:53.824919 6776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:53.856256 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.855765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.0.0.1:59214)
0929 16:00:53.856002 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.856096 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:36375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 16:00:53.857280 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.856933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.0.0.1:59216)
0929 16:00:53.857088 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.857155 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:36375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:00:53.917797 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.917206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.0.0.1:59226)
0929 16:00:53.917481 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.917605 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:36375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 16:00:53.918891 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.918542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.0.0.1:59234)
0929 16:00:53.918690 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.918780 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:36375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 16:00:53.940167 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 6651
I20260929 16:00:53.959743 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 6784
I20260929 16:00:53.973286 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.960530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.23.222.67:34437)
0929 16:00:53.960701 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:53.960708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:53.960733 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:00:53.960970 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:53.960976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:53.961057 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:53.961166 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:53.962482 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.963208 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.964558 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:53.964970 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:53.965103 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:53.967745 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:53.967762 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:53.967788 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:53.967825 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:53.970335 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.971065 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.971075 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.971078 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.971225 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:53.971667 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:53.971673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:53.971676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:53.972106 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:53.972423 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:53.972753 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 16:00:53.972936 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:00:53.974033 6908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:54.000083 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:53.999500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.0.0.1:59244)
0929 16:00:53.999739 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:53.999879 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:36375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 16:00:54.001302 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:54.000829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:36375 (local address 127.0.0.1:59248)
0929 16:00:54.001066 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:54.001169 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:36375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260929 16:00:54.063983 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 6784
I20260929 16:00:54.081406 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 6428
I20260929 16:00:54.082646 6491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:00:54.199024 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 6428
I20260929 16:00:54.224020 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42523
--webserver_interface=127.23.222.126
--webserver_port=43369
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:42523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:54.561831 6960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:54.562134 6960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:54.562311 6960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:54.569931 6960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:00:54.570050 6960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:54.570153 6960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:54.570255 6960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:00:54.570345 6960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:00:54.580894 6960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:42523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:42523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:54.582326 6960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:54.583750 6960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:54.595703 6965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:54.595837 6966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:54.595844 6968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:54.597100 6960 server_base.cc:1034] running on GCE node
I20260929 16:00:54.597990 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:54.599647 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:54.600908 6960 hybrid_clock.cc:648] HybridClock initialized: now 1790697654600848 us; error 65 us; skew 500 ppm
I20260929 16:00:54.601303 6960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:54 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697654, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:54.604832 6960 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:00:54.606562 6960 webserver.cc:460] Webserver started at http://127.23.222.126:43369/ using document root <none> and password file <none>
I20260929 16:00:54.607375 6960 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:54.607532 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:54.613369 6960 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 16:00:54.616328 6975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:54.617779 6960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:00:54.617969 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "29e858a095104ac4af470dba76519e8f"
format_stamp: "Formatted at 2026-09-29 16:00:48 on dist-test-slave-cq88"
server_key: "3d8e79b2f2efa33da8baa735e9320e3f"
server_key_iv: "cc886e18dfced2f30121963fcc4bba78"
server_key_version: "encryptionkey@0"
I20260929 16:00:54.619061 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:54.635808 6960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:54.640470 6960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:54.640789 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:54.660760 6960 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:42523
I20260929 16:00:54.660780 7027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:42523 every 8 connection(s)
I20260929 16:00:54.662320 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:00:54.667150 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 6960
I20260929 16:00:54.668195 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:36375
--local_ip_for_outbound_sockets=127.23.222.65
--tserver_master_addrs=127.23.222.126:42523
--webserver_port=34547
--webserver_interface=127.23.222.65
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:54.670563 7028 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:00:54.676400 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Bootstrap starting.
I20260929 16:00:54.698344 7028 log.cc:824] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:54.715374 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:00:54.715977 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f: Bootstrap complete.
I20260929 16:00:54.725879 7028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:54.726866 7028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e858a095104ac4af470dba76519e8f, State: Initialized, Role: FOLLOWER
I20260929 16:00:54.727609 7028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:54.727885 7028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:00:54.728025 7028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:00:54.728220 7028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:00:54.732329 7028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:54.732896 7028 leader_election.cc:302] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29e858a095104ac4af470dba76519e8f; no voters:
I20260929 16:00:54.733636 7028 leader_election.cc:288] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:00:54.733795 7032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:00:54.735036 7032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [term 3 LEADER]: Becoming Leader. State: Replica: 29e858a095104ac4af470dba76519e8f, State: Running, Role: LEADER
I20260929 16:00:54.735646 7032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } }
I20260929 16:00:54.736006 7028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:00:54.738483 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "29e858a095104ac4af470dba76519e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } } }
I20260929 16:00:54.738919 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:54.743988 7038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:00:54.744087 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29e858a095104ac4af470dba76519e8f. Latest consensus state: current_term: 3 leader_uuid: "29e858a095104ac4af470dba76519e8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e858a095104ac4af470dba76519e8f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 42523 } } }
I20260929 16:00:54.744479 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e858a095104ac4af470dba76519e8f [sys.catalog]: This master's current role is: LEADER
I20260929 16:00:54.753446 7038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:00:54.755074 7038 catalog_manager.cc:1259] Loaded cluster ID: 0f39dd2378e246c5a46967de256d16bf
I20260929 16:00:54.755276 7038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 16:00:54.757584 7038 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe287a0a9ad google::LogMessage::Fail() at ??:0
@ 0x7fe287a0d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe287a0a4de google::LogMessage::Flush() at ??:0
@ 0x7fe287a0ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe29d43d512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe29d43a976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe29d4c2610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe29d4c2439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe28a656345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe28a65991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe28a659799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe28a634eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe28d90e6db start_thread at ??:0
@ 0x7fe2868a461f clone at ??:0
W20260929 16:00:55.048969 7030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:55.049309 7030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:55.049479 7030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:55.057046 7030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:55.057205 7030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:55.057377 7030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:00:55.067404 7030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:36375
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=34547
--tserver_master_addrs=127.23.222.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:55.068714 7030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:55.070066 7030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:55.083949 7073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:55.083984 7070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:55.084008 7071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:55.084975 7030 server_base.cc:1034] running on GCE node
I20260929 16:00:55.085647 7030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:55.086694 7030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:55.087926 7030 hybrid_clock.cc:648] HybridClock initialized: now 1790697655087874 us; error 48 us; skew 500 ppm
I20260929 16:00:55.088281 7030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:55 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697655, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:55.091603 7030 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:00:55.093143 7030 webserver.cc:460] Webserver started at http://127.23.222.65:34547/ using document root <none> and password file <none>
I20260929 16:00:55.093838 7030 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:55.093984 7030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:55.099449 7030 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:00:55.104003 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:55.105338 7030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:00:55.105516 7030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "c22709258bbf47e9b5c50ddf62c3702c"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "842789e5f226299a63c44eb1176fef07"
server_key_iv: "3d2260844205cb813e3d6a6c0ccbbca7"
server_key_version: "encryptionkey@0"
I20260929 16:00:55.106504 7030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:55.131129 7030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:55.135962 7030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:55.136276 7030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:55.137331 7030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:55.139298 7087 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 16:00:55.145248 7030 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 16:00:55.145418 7030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260929 16:00:55.145571 7030 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 16:00:55.148813 7030 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 16:00:55.148927 7030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260929 16:00:55.149289 7087 tablet_bootstrap.cc:492] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Bootstrap starting.
I20260929 16:00:55.176090 7087 log.cc:824] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:00:55.190603 7087 tablet_bootstrap.cc:492] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:00:55.191196 7087 tablet_bootstrap.cc:492] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Bootstrap complete.
I20260929 16:00:55.191630 7030 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:36375
I20260929 16:00:55.191656 7196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:36375 every 8 connection(s)
I20260929 16:00:55.192698 7087 ts_tablet_manager.cc:1397] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.026s
I20260929 16:00:55.192958 7030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:00:55.195866 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 7030
I20260929 16:00:55.196655 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:42811
--local_ip_for_outbound_sockets=127.23.222.66
--tserver_master_addrs=127.23.222.126:42523
--webserver_port=43019
--webserver_interface=127.23.222.66
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:00:55.198480 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.196001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:42233)
0929 16:00:55.197238 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.197563 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":467,"threads_started":1}
W20260929 16:00:55.199227 7197 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:42523 (0 consecutive failures): Network error: Failed to ping master at 127.23.222.126:42523: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
I20260929 16:00:55.200174 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.199820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:46033)
0929 16:00:55.199983 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.200060 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:00:55.201112 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.200746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:54833)
0929 16:00:55.200894 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.200982 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:00:55.201412 7197 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:00:55.204221 7087 raft_consensus.cc:348] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:55.205492 7087 raft_consensus.cc:729] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c22709258bbf47e9b5c50ddf62c3702c, State: Initialized, Role: FOLLOWER
I20260929 16:00:55.206099 7087 consensus_queue.cc:260] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:55.207810 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.207448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:38835)
0929 16:00:55.207602 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.207690 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:00:55.208017 7087 ts_tablet_manager.cc:1428] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260929 16:00:55.209041 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.208580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:58015)
0929 16:00:55.208766 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.208876 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 16:00:55.532366 7200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:55.532666 7200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:55.532774 7200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:55.540082 7200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:55.540207 7200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:55.540323 7200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:00:55.550235 7200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:42811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=43019
--tserver_master_addrs=127.23.222.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:55.551625 7200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:55.553001 7200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:55.567890 7209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:55.567924 7206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:55.567962 7207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:55.568769 7200 server_base.cc:1034] running on GCE node
I20260929 16:00:55.569435 7200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:55.570421 7200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:55.571650 7200 hybrid_clock.cc:648] HybridClock initialized: now 1790697655571585 us; error 61 us; skew 500 ppm
I20260929 16:00:55.571986 7200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:55 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697655, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:55.575205 7200 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:00:55.576697 7200 webserver.cc:460] Webserver started at http://127.23.222.66:43019/ using document root <none> and password file <none>
I20260929 16:00:55.577344 7200 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:55.577445 7200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:55.582722 7200 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:00:55.585417 7216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:55.586737 7200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:00:55.586923 7200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "436d07338a374dfcb504c48028129c58"
format_stamp: "Formatted at 2026-09-29 16:00:49 on dist-test-slave-cq88"
server_key: "844b7a2772122ea52c4e4302236b21e0"
server_key_iv: "e12d93b0c3ccf06b39e976ad378879e9"
server_key_version: "encryptionkey@0"
I20260929 16:00:55.587966 7200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:55.622483 7200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:55.626770 7200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:55.627067 7200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:55.628131 7200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:55.629556 7200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:55.629644 7200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:55.629815 7200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:55.629905 7200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:55.664481 7200 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:42811
I20260929 16:00:55.664580 7331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:42811 every 8 connection(s)
I20260929 16:00:55.665980 7200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:00:55.670977 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.669103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:46705)
0929 16:00:55.669845 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.670084 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":275,"threads_started":1}
W20260929 16:00:55.671708 7332 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:42523 (0 consecutive failures): Network error: Failed to ping master at 127.23.222.126:42523: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
I20260929 16:00:55.672514 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.672161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:39645)
0929 16:00:55.672315 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.672385 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:00:55.673501 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:55.673150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:59717)
0929 16:00:55.673293 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:55.673386 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:00:55.673820 7332 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:00:55.674541 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 7200
I20260929 16:00:55.675155 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:34437
--local_ip_for_outbound_sockets=127.23.222.67
--tserver_master_addrs=127.23.222.126:42523
--webserver_port=36071
--webserver_interface=127.23.222.67
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:00:56.004109 7335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:00:56.004408 7335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:00:56.004504 7335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:00:56.012115 7335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:00:56.012247 7335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:00:56.012374 7335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:00:56.022523 7335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:34437
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=36071
--tserver_master_addrs=127.23.222.126:42523
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:00:56.023828 7335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:00:56.025175 7335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:00:56.038671 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:56.038585 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:00:56.038635 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:00:56.039609 7335 server_base.cc:1034] running on GCE node
I20260929 16:00:56.040277 7335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:00:56.041225 7335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:00:56.042464 7335 hybrid_clock.cc:648] HybridClock initialized: now 1790697656042419 us; error 59 us; skew 500 ppm
I20260929 16:00:56.042798 7335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:00:56 dist-test-slave-cq88 krb5kdc[5870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697656, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:00:56.046051 7335 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:00:56.047591 7335 webserver.cc:460] Webserver started at http://127.23.222.67:36071/ using document root <none> and password file <none>
I20260929 16:00:56.048285 7335 fs_manager.cc:359] Metadata directory not provided
I20260929 16:00:56.048446 7335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:00:56.054116 7335 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 16:00:56.057076 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:56.058585 7335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:00:56.058842 7335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "19f6ea2be6cb4dc499590ce2b5c8a896"
format_stamp: "Formatted at 2026-09-29 16:00:50 on dist-test-slave-cq88"
server_key: "257d5901640b32873a6b4eda331c80cc"
server_key_iv: "4404c6b7265617ecdc8f6a15dbdb5816"
server_key_version: "encryptionkey@0"
I20260929 16:00:56.060070 7335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:00:56.091337 7335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:00:56.096112 7335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:00:56.096421 7335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:00:56.097465 7335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:00:56.098896 7335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:00:56.099031 7335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:00:56.099184 7335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:00:56.099315 7335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 16:00:56.135124 7335 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:34437
I20260929 16:00:56.135182 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:34437 every 8 connection(s)
I20260929 16:00:56.136586 7335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:00:56.138765 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 7335
I20260929 16:00:56.141758 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.139816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:59671)
0929 16:00:56.140718 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:56.140926 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":324,"threads_started":1}
W20260929 16:00:56.142441 7464 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:42523 (0 consecutive failures): Network error: Failed to ping master at 127.23.222.126:42523: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
I20260929 16:00:56.143181 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.142874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:49703)
0929 16:00:56.143020 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:56.143081 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:00:56.144011 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.143687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:45189)
0929 16:00:56.143831 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:56.143908 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:00:56.144281 7464 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:00:56.210762 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.209976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:46089)
0929 16:00:56.210438 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:56.210524 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":224,"threads_started":1}
I20260929 16:00:56.596060 7470 raft_consensus.cc:482] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:00:56.596318 7470 raft_consensus.cc:504] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:56.597932 7470 leader_election.cc:288] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811), 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437)
Sep 29 16:00:56 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697655, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
Sep 29 16:00:56 dist-test-slave-cq88 krb5kdc[5870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697655, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 16:00:56.619052 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.598610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:57351 (local address 127.23.222.67:34437)
0929 16:00:56.599491 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:56.599504 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:56.599551 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:00:56.603775 (+ 4224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:56.603783 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:56.604113 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:56.604412 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:56.605301 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.606444 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.607701 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.608319 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.608504 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:56.611901 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:56.611907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:56.611920 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:56.611957 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:56.615184 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.616098 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.616107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.616116 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.616333 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.616773 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.616778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.616781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.617301 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:56.617496 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:56.618261 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 16:00:56.618635 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":557,"threads_started":1}
I20260929 16:00:56.619064 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.598345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:34437 (local address 127.23.222.65:57351)
0929 16:00:56.599457 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:56.599471 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:56.599621 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:56.604372 (+ 4751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:56.604377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:56.604823 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:56.605117 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:56.605126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.606606 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.606617 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:56.607560 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:56.607572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.608513 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.608520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:56.608683 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:56.609289 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:56.609306 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:56.611637 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:56.615393 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:56.615399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:56.615409 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:56.615957 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:56.616446 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:56.616451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:56.616455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:56.616652 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:56.617444 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:56.617452 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:56.617770 (+ 318us) client_negotiation.cc:769] Sending connection context
0929 16:00:56.618026 (+ 256us) client_negotiation.cc:240] Negotiation successful
0929 16:00:56.618548 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1014,"mutex_wait_us":41,"spinlock_wait_cycles":42240,"thread_start_us":222,"threads_started":1}
I20260929 16:00:56.619132 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.598417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:35539 (local address 127.23.222.66:42811)
0929 16:00:56.599214 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0929 16:00:56.599231 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 16:00:56.599310 (+ 79us) server_negotiation.cc:407] Connection header received
0929 16:00:56.601262 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:00:56.601271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:00:56.601664 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:00:56.601986 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:00:56.603874 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.605023 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.606638 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.607347 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.607543 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:56.611682 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:00:56.611690 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:00:56.611704 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:00:56.611747 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:00:56.615192 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.616085 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.616093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.616100 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.616338 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:00:56.616780 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:00:56.616785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:00:56.616788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:00:56.617330 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:00:56.617541 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 16:00:56.618270 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 16:00:56.618713 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":429,"threads_started":1}
I20260929 16:00:56.619138 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.598275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:42811 (local address 127.23.222.65:35539)
0929 16:00:56.598978 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:56.599012 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 16:00:56.599617 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:00:56.602104 (+ 2487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:00:56.602117 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:00:56.602982 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:00:56.603656 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:56.603685 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.605229 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.605232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:56.606442 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:00:56.606460 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:00:56.607548 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:00:56.607555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:00:56.607832 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:00:56.608527 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:00:56.608550 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:00:56.611376 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:00:56.615380 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:56.615389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:56.615404 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:56.615941 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:56.616456 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:00:56.616463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:00:56.616466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:00:56.616658 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:00:56.617472 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:00:56.617476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:00:56.617770 (+ 294us) client_negotiation.cc:769] Sending connection context
0929 16:00:56.618027 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 16:00:56.618742 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":179}
W20260929 16:00:56.622479 7083 leader_election.cc:341] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811): Not found: Tablet not found: ad399c6e5c054311a3ee83a63bcb4a82
W20260929 16:00:56.622764 7084 leader_election.cc:341] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437): Not found: Tablet not found: ad399c6e5c054311a3ee83a63bcb4a82
I20260929 16:00:56.622952 7084 leader_election.cc:302] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c22709258bbf47e9b5c50ddf62c3702c; no voters: 19f6ea2be6cb4dc499590ce2b5c8a896, 436d07338a374dfcb504c48028129c58
I20260929 16:00:56.623476 7470 raft_consensus.cc:2726] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:00:56.675343 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:56.674381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:36571)
0929 16:00:56.674884 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:56.675024 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":239,"threads_started":1}
I20260929 16:00:57.145615 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:57.144830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:36755)
0929 16:00:57.145286 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:57.145425 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260929 16:00:57.212384 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:57.211696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:59081)
0929 16:00:57.212125 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:57.212210 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":221,"threads_started":1}
I20260929 16:00:57.677034 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:57.676283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:59675)
0929 16:00:57.676761 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:57.676858 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":229,"threads_started":1}
I20260929 16:00:58.147173 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:58.146479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:40333)
0929 16:00:58.146913 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:58.146999 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
I20260929 16:00:58.214141 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:58.213270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:37449)
0929 16:00:58.213856 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:58.213953 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":289,"threads_started":1}
I20260929 16:00:58.529490 7480 raft_consensus.cc:482] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:00:58.529723 7480 raft_consensus.cc:504] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:00:58.530427 7480 leader_election.cc:288] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811), 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437)
W20260929 16:00:58.531572 7083 leader_election.cc:341] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811): Not found: Tablet not found: ad399c6e5c054311a3ee83a63bcb4a82
W20260929 16:00:58.531767 7084 leader_election.cc:341] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437): Not found: Tablet not found: ad399c6e5c054311a3ee83a63bcb4a82
I20260929 16:00:58.531870 7084 leader_election.cc:302] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c22709258bbf47e9b5c50ddf62c3702c; no voters: 19f6ea2be6cb4dc499590ce2b5c8a896, 436d07338a374dfcb504c48028129c58
I20260929 16:00:58.532084 7480 raft_consensus.cc:2726] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:00:58.678685 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:58.677996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:49815)
0929 16:00:58.678440 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:58.678522 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":246,"threads_started":1}
I20260929 16:00:59.148799 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:59.148078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:46713)
0929 16:00:59.148522 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:59.148607 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":240,"threads_started":1}
I20260929 16:00:59.215816 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:59.215095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:39879)
0929 16:00:59.215578 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:59.215660 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
I20260929 16:00:59.680447 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:00:59.679685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:38005)
0929 16:00:59.680152 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 16:00:59.680264 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260929 16:01:00.150482 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.149678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:44145)
0929 16:01:00.150193 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:00.150292 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
I20260929 16:01:00.218122 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.217209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:55545)
0929 16:01:00.217739 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:00.217929 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":296,"threads_started":1}
I20260929 16:01:00.682327 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:00.681498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:34751)
0929 16:01:00.682021 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:00.682134 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":263,"threads_started":1}
I20260929 16:01:01.152135 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:01.151412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:43117)
0929 16:01:01.151891 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:01.151977 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":222,"threads_started":1}
I20260929 16:01:01.219913 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:01.219111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:42691)
0929 16:01:01.219627 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:01.219714 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":302,"threads_started":1}
I20260929 16:01:01.527070 7490 raft_consensus.cc:482] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:01.527376 7490 raft_consensus.cc:504] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:01:01.528072 7490 leader_election.cc:288] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811), 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437)
W20260929 16:01:01.529137 7084 leader_election.cc:341] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437): Not found: Tablet not found: ad399c6e5c054311a3ee83a63bcb4a82
W20260929 16:01:01.529366 7083 leader_election.cc:341] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811): Not found: Tablet not found: ad399c6e5c054311a3ee83a63bcb4a82
I20260929 16:01:01.529529 7083 leader_election.cc:302] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c22709258bbf47e9b5c50ddf62c3702c; no voters: 19f6ea2be6cb4dc499590ce2b5c8a896, 436d07338a374dfcb504c48028129c58
I20260929 16:01:01.529827 7490 raft_consensus.cc:2726] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:01:01.684206 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:01.683463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:52405)
0929 16:01:01.683943 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:01.684028 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":257,"threads_started":1}
I20260929 16:01:02.153767 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:02.153030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:42209)
0929 16:01:02.153514 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:02.153595 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":269,"threads_started":1}
I20260929 16:01:02.221583 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:02.220861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:58541)
0929 16:01:02.221305 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:02.221387 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":242,"threads_started":1}
I20260929 16:01:02.686000 7494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:02.685180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:49183)
0929 16:01:02.685727 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:02.685813 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":255,"threads_started":1}
I20260929 16:01:03.155719 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:03.154801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:49115)
0929 16:01:03.155406 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:03.155492 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":263,"threads_started":1}
I20260929 16:01:03.223716 7496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:03.222704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:56651)
0929 16:01:03.223355 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:03.223491 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":364,"threads_started":1}
I20260929 16:01:03.687687 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:03.686943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:41463)
0929 16:01:03.687418 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:03.687503 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":228,"threads_started":1}
I20260929 16:01:04.157442 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.156762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:36969)
0929 16:01:04.157179 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.157262 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":213,"threads_started":1}
I20260929 16:01:04.225971 7499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.224982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:56873)
0929 16:01:04.225627 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.225751 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":348,"threads_started":1}
I20260929 16:01:04.689555 7500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:04.688792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:33093)
0929 16:01:04.689295 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:04.689382 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":237,"threads_started":1}
I20260929 16:01:05.159211 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.158483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:40245)
0929 16:01:05.158959 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.159048 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":222,"threads_started":1}
I20260929 16:01:05.227808 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.227059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:42917)
0929 16:01:05.227554 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.227642 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":277,"threads_started":1}
I20260929 16:01:05.691427 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:05.690595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:37203)
0929 16:01:05.691070 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:05.691165 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":300,"threads_started":1}
I20260929 16:01:05.963325 7504 raft_consensus.cc:482] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:05.963577 7504 raft_consensus.cc:504] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c22709258bbf47e9b5c50ddf62c3702c" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 36375 } } peers { permanent_uuid: "436d07338a374dfcb504c48028129c58" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 42811 } } peers { permanent_uuid: "19f6ea2be6cb4dc499590ce2b5c8a896" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 34437 } }
I20260929 16:01:05.964460 7504 leader_election.cc:288] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811), 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437)
W20260929 16:01:05.965437 7084 leader_election.cc:341] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 19f6ea2be6cb4dc499590ce2b5c8a896 (127.23.222.67:34437): Not found: Tablet not found: ad399c6e5c054311a3ee83a63bcb4a82
W20260929 16:01:05.965644 7083 leader_election.cc:341] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 436d07338a374dfcb504c48028129c58 (127.23.222.66:42811): Not found: Tablet not found: ad399c6e5c054311a3ee83a63bcb4a82
I20260929 16:01:05.965813 7083 leader_election.cc:302] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c22709258bbf47e9b5c50ddf62c3702c; no voters: 19f6ea2be6cb4dc499590ce2b5c8a896, 436d07338a374dfcb504c48028129c58
I20260929 16:01:05.966120 7504 raft_consensus.cc:2726] T ad399c6e5c054311a3ee83a63bcb4a82 P c22709258bbf47e9b5c50ddf62c3702c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 16:01:06.160935 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:06.160153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:49901)
0929 16:01:06.160647 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:06.160744 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":276,"threads_started":1}
I20260929 16:01:06.229736 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:06.228842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:51743)
0929 16:01:06.229385 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:06.229528 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":284,"threads_started":1}
I20260929 16:01:06.693303 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:06.692456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:50175)
0929 16:01:06.693013 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:06.693119 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":300,"threads_started":1}
I20260929 16:01:07.162570 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:07.161825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:43515)
0929 16:01:07.162309 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:07.162398 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":252,"threads_started":1}
I20260929 16:01:07.231597 7509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:07.230752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:59407)
0929 16:01:07.231280 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:07.231385 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":261,"threads_started":1}
I20260929 16:01:07.695422 7510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:07.694384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:39467)
0929 16:01:07.695040 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:07.695154 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":338,"threads_started":1}
I20260929 16:01:08.164405 7511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:08.163560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:49345)
0929 16:01:08.164121 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:08.164207 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":284,"threads_started":1}
I20260929 16:01:08.233362 7512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:08.232608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:39859)
0929 16:01:08.233089 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:08.233183 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":247,"threads_started":1}
I20260929 16:01:08.697194 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:08.696441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:59863)
0929 16:01:08.696913 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:08.697017 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
I20260929 16:01:09.166034 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:09.165284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:44323)
0929 16:01:09.165726 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:09.165859 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":220,"threads_started":1}
I20260929 16:01:09.235177 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:09.234377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:58877)
0929 16:01:09.234889 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:09.234977 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":283,"threads_started":1}
I20260929 16:01:09.699079 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:09.698252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:38773)
0929 16:01:09.698789 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:09.698882 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":266,"threads_started":1}
I20260929 16:01:10.167714 7517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:10.166946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:43495)
0929 16:01:10.167427 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:10.167524 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":225,"threads_started":1}
I20260929 16:01:10.237192 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:10.236464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.65:36259)
0929 16:01:10.236913 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:10.237006 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":217,"threads_started":1}
I20260929 16:01:10.701122 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:10.700372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.66:42381)
0929 16:01:10.700862 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:10.700957 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":229,"threads_started":1}
I20260929 16:01:11.143095 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 7030
I20260929 16:01:11.159924 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:11.144342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.23.222.65:36375)
0929 16:01:11.145250 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:11.145261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:11.145300 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:01:11.145404 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:11.145411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:11.145729 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:11.145960 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:11.147562 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:11.148660 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:11.150122 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:11.150859 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:11.150983 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:11.153760 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:11.153773 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:11.153787 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:11.153829 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:11.157048 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:11.157787 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:11.157795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:11.157798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:11.157955 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:11.158333 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:11.158337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:11.158339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:11.158738 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:11.159022 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:11.159376 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 16:01:11.159607 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":539,"threads_started":1}
I20260929 16:01:11.161252 7192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:11.169726 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:11.168952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:42523 (local address 127.23.222.67:55423)
0929 16:01:11.169447 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:11.169555 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:42523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":242,"threads_started":1}
I20260929 16:01:11.260053 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 7030
I20260929 16:01:11.277642 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 7200
I20260929 16:01:11.292194 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:11.278446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.23.222.66:42811)
0929 16:01:11.278958 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:11.278962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:11.278979 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:01:11.279034 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:11.279037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:11.279095 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:11.279186 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:11.280653 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:11.281483 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:11.282915 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:11.283469 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:11.283573 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:11.286362 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:11.286367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:11.286371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:11.286409 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:11.289272 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:11.290089 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:11.290093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:11.290096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:11.290249 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:11.290594 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:11.290598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:11.290600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:11.290923 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:11.291204 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:11.291658 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 16:01:11.291856 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":293,"threads_started":1}
I20260929 16:01:11.293035 7327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:11.377616 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 7200
I20260929 16:01:11.396366 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 7335
I20260929 16:01:11.411047 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:11.397297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.23.222.67:34437)
0929 16:01:11.397953 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:11.397958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:11.397976 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:01:11.398031 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:11.398035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:11.398097 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:11.398200 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:11.400034 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:11.400948 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:11.402243 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:11.402689 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:11.402823 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:11.405487 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:11.405492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:11.405497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:11.405540 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:11.408522 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:11.409241 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:11.409246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:11.409248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:11.409404 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:11.409765 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:11.409769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:11.409771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:11.410053 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:11.410270 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:11.410533 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 16:01:11.410697 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":364,"threads_started":1}
I20260929 16:01:11.411772 7459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:11.495143 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 7335
2026-09-29T16:01:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24517 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:01:11 dist-test-slave-cq88 krb5kdc[7530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:01:11 dist-test-slave-cq88 krb5kdc[7530](info): set up 2 sockets
Sep 29 16:01:11 dist-test-slave-cq88 krb5kdc[7530](info): commencing operation
krb5kdc: starting...
W20260929 16:01:12.602880 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:01:12 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:01:12.858886 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37599
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:37599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:13.191083 7543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:13.191368 7543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:13.191556 7543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:13.198681 7543 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:01:13.198776 7543 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:01:13.198853 7543 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:01:13.198931 7543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:13.199024 7543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:13.199157 7543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:13.199208 7543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:13.199325 7543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:13.209271 7543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:37599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:01:13.210620 7543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:13.212033 7543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:13.227509 7549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:13.227587 7551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:13.227546 7548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:13.228566 7543 server_base.cc:1034] running on GCE node
I20260929 16:01:13.229434 7543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:13.230950 7543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:13.232215 7543 hybrid_clock.cc:648] HybridClock initialized: now 1790697673232138 us; error 80 us; skew 500 ppm
I20260929 16:01:13.232578 7543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:13 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:13.240495 7543 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:01:13.242110 7543 webserver.cc:460] Webserver started at http://127.23.222.126:42817/ using document root <none> and password file <none>
I20260929 16:01:13.242841 7543 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:13.242974 7543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:13.243384 7543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:13.246433 7543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "73588ebc21114361a4f6fe399cc48cdd"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "7f296754d45deaceb3c5ca4a2f093d7e"
server_key_iv: "bc6afcbf64f3ae71852dfbedd5a9bcb2"
server_key_version: "encryptionkey@0"
I20260929 16:01:13.247223 7543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "73588ebc21114361a4f6fe399cc48cdd"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "7f296754d45deaceb3c5ca4a2f093d7e"
server_key_iv: "bc6afcbf64f3ae71852dfbedd5a9bcb2"
server_key_version: "encryptionkey@0"
I20260929 16:01:13.252465 7543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 16:01:13.256078 7558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:13.257364 7543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:01:13.257543 7543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "73588ebc21114361a4f6fe399cc48cdd"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "7f296754d45deaceb3c5ca4a2f093d7e"
server_key_iv: "bc6afcbf64f3ae71852dfbedd5a9bcb2"
server_key_version: "encryptionkey@0"
I20260929 16:01:13.257764 7543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:13.284200 7543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:13.288789 7543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:13.289114 7543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:13.308485 7543 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:37599
I20260929 16:01:13.308511 7609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:37599 every 8 connection(s)
I20260929 16:01:13.309918 7543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:01:13.313150 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 7543
I20260929 16:01:13.313403 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:01:13.314077 24441 external_mini_cluster.cc:1428] Setting key 55034d7efe77c0e499efe06005231754
I20260929 16:01:13.314244 7610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:13.329567 7610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Bootstrap starting.
Sep 29 16:01:13 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:01:13.332923 7610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:13.334348 7610 log.cc:824] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:13.338012 7610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: No bootstrap required, opened a new log
I20260929 16:01:13.340121 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.316259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.23.222.126:37599)
0929 16:01:13.317105 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.317120 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.317162 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:01:13.319105 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.319138 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.319668 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.320017 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:13.321991 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.323413 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.325363 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.326086 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.326230 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.331336 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:13.331350 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:13.331365 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:13.331413 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:13.335230 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:13.336179 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:13.336187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:13.336198 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:13.336420 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:13.336800 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:13.336804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:13.336807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:13.337503 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:13.337778 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.338250 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.338743 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":382,"threads_started":1}
I20260929 16:01:13.347159 7610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:01:13.347643 7610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:13.347723 7610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73588ebc21114361a4f6fe399cc48cdd, State: Initialized, Role: FOLLOWER
I20260929 16:01:13.348366 7610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:01:13.348591 7610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:01:13.348690 7610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:01:13.348851 7610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:13.351409 7610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:01:13.351891 7610 leader_election.cc:302] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73588ebc21114361a4f6fe399cc48cdd; no voters:
I20260929 16:01:13.352488 7610 leader_election.cc:288] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:01:13.352725 7616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:13.353459 7616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 1 LEADER]: Becoming Leader. State: Replica: 73588ebc21114361a4f6fe399cc48cdd, State: Running, Role: LEADER
I20260929 16:01:13.354035 7616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:01:13.354832 7610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:01:13.356909 7618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73588ebc21114361a4f6fe399cc48cdd. Latest consensus state: current_term: 1 leader_uuid: "73588ebc21114361a4f6fe399cc48cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } } }
I20260929 16:01:13.357241 7618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:13.361094 7617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73588ebc21114361a4f6fe399cc48cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } } }
I20260929 16:01:13.361549 7617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:13.362079 7622 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:01:13.368443 7622 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:01:13.378165 7622 catalog_manager.cc:1347] Generated new cluster ID: d6e40febac324fffa2e863424f1506b8
I20260929 16:01:13.378301 7622 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:01:13.402596 7622 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:01:13.404608 7622 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:01:13.414399 7622 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Generated new TSK 0
I20260929 16:01:13.415405 7622 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 16:01:13.490646 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37599
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:13.855715 7637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:13.856164 7637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:13.856386 7637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:13.864363 7637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:13.864473 7637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:13.864586 7637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:01:13.874835 7637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:01:13.876168 7637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:13.877499 7637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:13.892057 7642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:13.892149 7643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:13.892104 7645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:13.893416 7637 server_base.cc:1034] running on GCE node
I20260929 16:01:13.894109 7637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:13.895195 7637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:13.896507 7637 hybrid_clock.cc:648] HybridClock initialized: now 1790697673896455 us; error 41 us; skew 500 ppm
I20260929 16:01:13.896873 7637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:13 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:13.900676 7637 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:01:13.902403 7637 webserver.cc:460] Webserver started at http://127.23.222.65:32789/ using document root <none> and password file <none>
I20260929 16:01:13.903148 7637 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:13.903344 7637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:13.903705 7637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:13.906849 7637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "fb59afc1f1bf44bea2e2cfc3b651b845"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "22b536b1cc08478f648d03c391d33604"
server_key_iv: "5774bca8c059842e9eebe0b50dac0be0"
server_key_version: "encryptionkey@0"
I20260929 16:01:13.907704 7637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "fb59afc1f1bf44bea2e2cfc3b651b845"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "22b536b1cc08478f648d03c391d33604"
server_key_iv: "5774bca8c059842e9eebe0b50dac0be0"
server_key_version: "encryptionkey@0"
I20260929 16:01:13.913197 7637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:01:13.916980 7652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:13.918273 7637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:01:13.918452 7637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "fb59afc1f1bf44bea2e2cfc3b651b845"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "22b536b1cc08478f648d03c391d33604"
server_key_iv: "5774bca8c059842e9eebe0b50dac0be0"
server_key_version: "encryptionkey@0"
I20260929 16:01:13.918643 7637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:13.933007 7637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:13.937165 7637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:13.937481 7637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:13.938467 7637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:13.939838 7637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:13.939952 7637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:13.940104 7637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:13.940181 7637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:13.975099 7637 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:42973
I20260929 16:01:13.975114 7765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:42973 every 8 connection(s)
I20260929 16:01:13.976444 7637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:01:13.978891 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 7637
I20260929 16:01:13.979123 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 16:01:13.979931 24441 external_mini_cluster.cc:1428] Setting key 089f1c9be6226da54ea729e9bbf91c2e
Sep 29 16:01:13 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697673, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:01:13.999871 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.979266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:44087 (local address 127.23.222.126:37599)
0929 16:01:13.979788 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:13.979795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:13.981080 (+ 1285us) server_negotiation.cc:407] Connection header received
0929 16:01:13.983257 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:13.983265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:13.983377 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:13.983503 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:13.985175 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.986374 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.988117 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.988844 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.989000 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.992567 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:13.992576 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:13.992584 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:13.992633 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:13.996189 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:13.997089 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:13.997097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:13.997104 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:13.997257 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:13.997767 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:13.997775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:13.997781 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:13.998165 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:13.998334 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:13.999043 (+ 709us) server_negotiation.cc:299] Negotiation successful
0929 16:01:13.999361 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":256,"threads_started":1}
I20260929 16:01:14.000581 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:13.979998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:44087)
0929 16:01:13.980911 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:13.980963 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:13.982920 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:13.983736 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:13.983745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:13.984337 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:13.984969 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.984989 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.986556 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.986560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.987957 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:13.987970 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:13.989061 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:13.989066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:13.989271 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:13.990001 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:13.990024 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:13.992324 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:13.996391 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:13.996398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:13.996412 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:13.996933 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:13.997403 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:13.997407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:13.997409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:13.997630 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:13.998291 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:13.998297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:13.998638 (+ 341us) client_negotiation.cc:769] Sending connection context
0929 16:01:13.998888 (+ 250us) client_negotiation.cc:240] Negotiation successful
0929 16:01:13.999394 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":373,"threads_started":1}
I20260929 16:01:14.002331 7766 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37599
I20260929 16:01:14.002725 7766 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:14.003739 7766 heartbeater.cc:511] Master 127.23.222.126:37599 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
I20260929 16:01:14.006223 7575 ts_manager.cc:194] Registered new tserver with Master: fb59afc1f1bf44bea2e2cfc3b651b845 (127.23.222.65:42973)
I20260929 16:01:14.007966 7575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:44087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 16:01:14.038890 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37599
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:14.374104 7772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:14.374388 7772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:14.374565 7772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:14.381831 7772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:14.381968 7772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:14.382164 7772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:01:14.391758 7772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:01:14.393076 7772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:14.394367 7772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:14.408188 7778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:14.408205 7780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:14.408468 7777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:14.409042 7772 server_base.cc:1034] running on GCE node
I20260929 16:01:14.409593 7772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:14.410483 7772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:14.411671 7772 hybrid_clock.cc:648] HybridClock initialized: now 1790697674411613 us; error 53 us; skew 500 ppm
I20260929 16:01:14.412035 7772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:14 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697674, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:14.415319 7772 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:01:14.416796 7772 webserver.cc:460] Webserver started at http://127.23.222.66:42911/ using document root <none> and password file <none>
I20260929 16:01:14.417510 7772 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:14.417655 7772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:14.417981 7772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:14.420971 7772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "fc4712fa936540f5b0728b87379eca37"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "42aeef3c5ce0e819beb3a2b84423a9c0"
server_key_iv: "1196a270605030c061dbd36450a35526"
server_key_version: "encryptionkey@0"
I20260929 16:01:14.421800 7772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "fc4712fa936540f5b0728b87379eca37"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "42aeef3c5ce0e819beb3a2b84423a9c0"
server_key_iv: "1196a270605030c061dbd36450a35526"
server_key_version: "encryptionkey@0"
I20260929 16:01:14.427325 7772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:01:14.431157 7787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:14.432664 7772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:01:14.432852 7772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "fc4712fa936540f5b0728b87379eca37"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "42aeef3c5ce0e819beb3a2b84423a9c0"
server_key_iv: "1196a270605030c061dbd36450a35526"
server_key_version: "encryptionkey@0"
I20260929 16:01:14.433075 7772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:14.454581 7772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:14.458890 7772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:14.459192 7772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:14.460176 7772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:14.461607 7772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:14.461712 7772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:14.461859 7772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:14.461932 7772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:14.495817 7772 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:40511
I20260929 16:01:14.495878 7899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:40511 every 8 connection(s)
I20260929 16:01:14.497197 7772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:01:14.502542 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 7772
I20260929 16:01:14.502743 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 16:01:14.503337 24441 external_mini_cluster.cc:1428] Setting key 6884c51676cac233949988926e0983ea
Sep 29 16:01:14 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697674, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:01:14.521003 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:14.499945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:58883 (local address 127.23.222.126:37599)
0929 16:01:14.500145 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:14.500150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:14.501320 (+ 1170us) server_negotiation.cc:407] Connection header received
0929 16:01:14.504005 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:14.504012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:14.504110 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:14.504220 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:14.506131 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.506838 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.509528 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.510043 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.510156 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:14.514784 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:14.514790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:14.514793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:14.514831 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:14.517645 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:14.518472 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:14.518477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:14.518479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:14.518613 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:14.519087 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:14.519091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:14.519093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:14.519452 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:14.519615 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:14.520375 (+ 760us) server_negotiation.cc:299] Negotiation successful
0929 16:01:14.520629 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:01:14.522162 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:14.500439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:58883)
0929 16:01:14.501149 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:14.501192 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:14.503654 (+ 2462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:14.504525 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:14.504537 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:14.505114 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:14.505938 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:14.505965 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.507011 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.507019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:14.509327 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:14.509346 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:14.510312 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:14.510322 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:14.510700 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:14.511618 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:14.511654 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:14.514486 (+ 2832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:14.517811 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:14.517821 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:14.517840 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:14.518311 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:14.518737 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:14.518742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:14.518746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:14.518950 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:14.519587 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:14.519596 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:14.519983 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 16:01:14.520266 (+ 283us) client_negotiation.cc:240] Negotiation successful
0929 16:01:14.520856 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":269,"threads_started":1}
I20260929 16:01:14.523826 7900 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37599
I20260929 16:01:14.524230 7900 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:14.525252 7900 heartbeater.cc:511] Master 127.23.222.126:37599 requested a full tablet report, sending...
I20260929 16:01:14.527213 7575 ts_manager.cc:194] Registered new tserver with Master: fc4712fa936540f5b0728b87379eca37 (127.23.222.66:40511)
I20260929 16:01:14.528537 7575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:58883
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 16:01:14.563263 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37599
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:14.894969 7906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:14.895378 7906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:14.895562 7906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:14.903020 7906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:14.903144 7906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:14.903369 7906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:01:14.913307 7906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:01:14.914678 7906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:14.916088 7906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:14.929261 7911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:14.929519 7914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:14.929287 7912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:14.930083 7906 server_base.cc:1034] running on GCE node
I20260929 16:01:14.930712 7906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:14.931738 7906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:14.932976 7906 hybrid_clock.cc:648] HybridClock initialized: now 1790697674932918 us; error 49 us; skew 500 ppm
I20260929 16:01:14.933305 7906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:14 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697674, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:14.936784 7906 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:01:14.938292 7906 webserver.cc:460] Webserver started at http://127.23.222.67:33269/ using document root <none> and password file <none>
I20260929 16:01:14.938941 7906 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:14.939042 7906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:14.939389 7906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:01:14.942196 7906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "1a49823299094d45a46d74b35452764c"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "9d2f009bb6f70fc941b52c8933ce3a18"
server_key_iv: "313197c773242466086ea7c0b5063580"
server_key_version: "encryptionkey@0"
I20260929 16:01:14.942891 7906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "1a49823299094d45a46d74b35452764c"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "9d2f009bb6f70fc941b52c8933ce3a18"
server_key_iv: "313197c773242466086ea7c0b5063580"
server_key_version: "encryptionkey@0"
I20260929 16:01:14.947962 7906 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 16:01:14.951216 7921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:14.952618 7906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:01:14.952761 7906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "1a49823299094d45a46d74b35452764c"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "9d2f009bb6f70fc941b52c8933ce3a18"
server_key_iv: "313197c773242466086ea7c0b5063580"
server_key_version: "encryptionkey@0"
I20260929 16:01:14.952944 7906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:14.991931 7906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:14.996455 7906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:14.996728 7906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:14.997807 7906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:14.999262 7906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:14.999346 7906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 16:01:14.999431 7906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:14.999476 7906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:15.011405 7766 heartbeater.cc:503] Master 127.23.222.126:37599 was elected leader, sending a full tablet report...
I20260929 16:01:15.035282 7906 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:33435
I20260929 16:01:15.035349 8033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:33435 every 8 connection(s)
I20260929 16:01:15.036792 7906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:01:15.038153 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 7906
I20260929 16:01:15.038364 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 16:01:15.038940 24441 external_mini_cluster.cc:1428] Setting key b7052ab19cdd25e36b9f06a319e41032
Sep 29 16:01:15 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697674, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:01:15.059119 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.039863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:33531 (local address 127.23.222.126:37599)
0929 16:01:15.040589 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.040597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.041524 (+ 927us) server_negotiation.cc:407] Connection header received
0929 16:01:15.043750 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.043755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.043859 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.044003 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:15.045640 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.046451 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.048100 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.048524 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.048671 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.052184 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:15.052195 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:15.052198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:15.052235 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:15.055654 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.056618 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.056622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.056625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.056769 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.057209 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.057213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.057215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.057608 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:15.057752 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.058612 (+ 860us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.058827 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":323,"threads_started":1}
I20260929 16:01:15.060138 8037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.040540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:33531)
0929 16:01:15.041370 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:15.041411 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:15.043470 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:15.044256 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:15.044265 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:15.044854 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:15.045455 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.045483 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.046607 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.046612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.047951 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.047965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.048770 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.048779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.049018 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.049669 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:15.049691 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:15.051949 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:15.055788 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:15.055795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:15.055810 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:15.056437 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:15.056885 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:15.056888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:15.056890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:15.057103 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:15.057750 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:15.057756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:15.058183 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 16:01:15.058484 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 16:01:15.058907 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":315,"threads_started":1}
I20260929 16:01:15.061699 8034 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37599
I20260929 16:01:15.062094 8034 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:15.062985 8034 heartbeater.cc:511] Master 127.23.222.126:37599 requested a full tablet report, sending...
I20260929 16:01:15.064705 7575 ts_manager.cc:194] Registered new tserver with Master: 1a49823299094d45a46d74b35452764c (127.23.222.67:33435)
I20260929 16:01:15.065652 7575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:33531
I20260929 16:01:15.067799 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:01:15.085368 8036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.072453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.23.222.126:37599)
0929 16:01:15.072629 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.072634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.073034 (+ 400us) server_negotiation.cc:407] Connection header received
0929 16:01:15.073193 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.073197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.073299 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.073436 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:15.075294 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.076028 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.077341 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.077754 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.077862 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.080260 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:15.080265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:15.080268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:15.080298 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:15.082762 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.083549 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.083553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.083555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.083676 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.084039 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.084043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.084045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.084392 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:15.084591 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.084932 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.085115 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:01:15.089879 7574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:01:15.096824 7574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:01:15.118945 8048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.109378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.23.222.65:42973)
0929 16:01:15.110686 (+ 1308us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.110695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.110732 (+ 37us) server_negotiation.cc:407] Connection header received
0929 16:01:15.110844 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.110854 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.111118 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.112121 (+ 1003us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:15.112392 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.114000 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.116785 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.117759 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.117917 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.118166 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.118422 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.118590 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":965,"threads_started":1}
I20260929 16:01:15.119031 8047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.109082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:42973 (local address 127.0.0.1:52564)
0929 16:01:15.109797 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:15.109844 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:15.110040 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:15.111389 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:15.111401 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:15.111444 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:15.111888 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.111906 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.114140 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.114146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.116607 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.116623 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.117977 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.117983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.118172 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.118230 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 16:01:15.118386 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 16:01:15.118584 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":2,"thread_start_us":296,"threads_started":1}
I20260929 16:01:15.120707 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.110423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:40511 (local address 127.0.0.1:34234)
0929 16:01:15.112211 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:15.112229 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:15.112337 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:15.113174 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:15.113179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:15.113203 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:15.113566 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.113576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.115673 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.115678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.117894 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.117915 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.119901 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.119908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.120095 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.120129 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:01:15.120196 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 16:01:15.120329 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1674,"mutex_wait_us":24,"thread_start_us":337,"threads_started":1}
I20260929 16:01:15.121752 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.110300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:33435 (local address 127.0.0.1:60916)
0929 16:01:15.111049 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:15.111066 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:15.111179 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:15.114621 (+ 3442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:15.114625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:15.114649 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:15.115094 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.115106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.116932 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.116938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.119096 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.119111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.120991 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.121000 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.121161 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.121191 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:01:15.121255 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 16:01:15.121393 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":3,"thread_start_us":1615,"threads_started":1}
I20260929 16:01:15.121793 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.111149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.23.222.66:40511)
0929 16:01:15.111710 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.111721 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.112467 (+ 746us) server_negotiation.cc:407] Connection header received
0929 16:01:15.112573 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.112579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.112789 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.112969 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:15.113758 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.115367 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.118088 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.119661 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.121067 (+ 1406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.121257 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.121343 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.121506 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260929 16:01:15.122931 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.111116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60916 (local address 127.23.222.67:33435)
0929 16:01:15.114172 (+ 3056us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.114177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.114201 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:01:15.114269 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.114273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.114426 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.114630 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:15.115292 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.116791 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.119319 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.120756 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.122130 (+ 1374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.122335 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.122411 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.122570 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2878,"thread_start_us":2722,"threads_started":1}
I20260929 16:01:15.123416 7700 tablet_service.cc:1467] Processing CreateTablet for tablet 76144ddeb82b4e2d9f9ef217391d9d23 (DEFAULT_TABLE table=test-table [id=2d77d4a987da4392abc73a5c5d9ee5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:15.125017 7700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76144ddeb82b4e2d9f9ef217391d9d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:15.125370 7835 tablet_service.cc:1467] Processing CreateTablet for tablet 76144ddeb82b4e2d9f9ef217391d9d23 (DEFAULT_TABLE table=test-table [id=2d77d4a987da4392abc73a5c5d9ee5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:15.126784 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76144ddeb82b4e2d9f9ef217391d9d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:15.130070 7969 tablet_service.cc:1467] Processing CreateTablet for tablet 76144ddeb82b4e2d9f9ef217391d9d23 (DEFAULT_TABLE table=test-table [id=2d77d4a987da4392abc73a5c5d9ee5d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:15.131517 7969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76144ddeb82b4e2d9f9ef217391d9d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:15.141203 8054 tablet_bootstrap.cc:492] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: Bootstrap starting.
I20260929 16:01:15.142027 8053 tablet_bootstrap.cc:492] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: Bootstrap starting.
I20260929 16:01:15.143196 8055 tablet_bootstrap.cc:492] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: Bootstrap starting.
I20260929 16:01:15.145483 8053 tablet_bootstrap.cc:654] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:15.146514 8053 log.cc:824] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:15.146983 8055 tablet_bootstrap.cc:654] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:15.146981 8054 tablet_bootstrap.cc:654] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:15.148365 8055 log.cc:824] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:15.148370 8054 log.cc:824] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:15.150701 8053 tablet_bootstrap.cc:492] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: No bootstrap required, opened a new log
I20260929 16:01:15.151032 8053 ts_tablet_manager.cc:1397] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260929 16:01:15.151698 8055 tablet_bootstrap.cc:492] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: No bootstrap required, opened a new log
I20260929 16:01:15.152006 8055 ts_tablet_manager.cc:1397] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:01:15.153311 8054 tablet_bootstrap.cc:492] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: No bootstrap required, opened a new log
I20260929 16:01:15.153610 8054 ts_tablet_manager.cc:1397] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260929 16:01:15.159695 8053 raft_consensus.cc:348] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:15.160106 8053 raft_consensus.cc:374] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:15.160238 8053 raft_consensus.cc:729] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59afc1f1bf44bea2e2cfc3b651b845, State: Initialized, Role: FOLLOWER
I20260929 16:01:15.160924 8053 consensus_queue.cc:260] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:15.163010 8053 ts_tablet_manager.cc:1428] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 16:01:15.163554 8055 raft_consensus.cc:348] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:15.164095 8055 raft_consensus.cc:374] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:15.164216 8055 raft_consensus.cc:729] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a49823299094d45a46d74b35452764c, State: Initialized, Role: FOLLOWER
I20260929 16:01:15.164894 8055 consensus_queue.cc:260] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:15.165053 8054 raft_consensus.cc:348] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:15.165659 8054 raft_consensus.cc:374] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:15.165827 8054 raft_consensus.cc:729] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4712fa936540f5b0728b87379eca37, State: Initialized, Role: FOLLOWER
I20260929 16:01:15.166126 8034 heartbeater.cc:503] Master 127.23.222.126:37599 was elected leader, sending a full tablet report...
I20260929 16:01:15.166883 8055 ts_tablet_manager.cc:1428] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260929 16:01:15.167774 8054 consensus_queue.cc:260] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:15.169417 7900 heartbeater.cc:503] Master 127.23.222.126:37599 was elected leader, sending a full tablet report...
I20260929 16:01:15.170276 8054 ts_tablet_manager.cc:1428] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260929 16:01:15.228922 7767 tablet.cc:2369] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:01:15.249285 7901 tablet.cc:2369] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:01:15.288594 8035 tablet.cc:2369] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:15.595485 8059 raft_consensus.cc:482] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:15.595760 8059 raft_consensus.cc:504] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:15.597453 8059 leader_election.cc:288] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a49823299094d45a46d74b35452764c (127.23.222.67:33435), fc4712fa936540f5b0728b87379eca37 (127.23.222.66:40511)
I20260929 16:01:15.604028 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.597755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:59881 (local address 127.23.222.67:33435)
0929 16:01:15.597986 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.597995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.598721 (+ 726us) server_negotiation.cc:407] Connection header received
0929 16:01:15.599001 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.599006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.599067 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.599166 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:15.599746 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.600443 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.602278 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.603044 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.603233 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.603344 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.603621 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.603748 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:01:15.604205 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.598122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:56483 (local address 127.23.222.66:40511)
0929 16:01:15.598383 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.598392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.599048 (+ 656us) server_negotiation.cc:407] Connection header received
0929 16:01:15.599288 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.599294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.599365 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.599466 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:15.600016 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.600719 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.602444 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.603223 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.603454 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.603538 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.603806 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.603930 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 16:01:15.604274 8063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.597952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:40511 (local address 127.23.222.65:56483)
0929 16:01:15.598916 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:15.598929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:15.599095 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:15.599484 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:15.599489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:15.599512 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:15.599862 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.599871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.600867 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.600872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.602289 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.602301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.603491 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.603497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.603650 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.603673 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:01:15.603779 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 16:01:15.603910 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":615,"threads_started":1}
I20260929 16:01:15.604554 8062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.597681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:33435 (local address 127.23.222.65:59881)
0929 16:01:15.598582 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:15.598612 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:15.598812 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:15.599191 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:15.599195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:15.599222 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:15.599589 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.599598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.600569 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.600577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.602123 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:15.602135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.603281 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.603287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:15.603462 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.603490 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:01:15.603597 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 16:01:15.603730 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":43,"thread_start_us":570,"threads_started":1}
I20260929 16:01:15.605481 7989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76144ddeb82b4e2d9f9ef217391d9d23" candidate_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a49823299094d45a46d74b35452764c" is_pre_election: true
I20260929 16:01:15.605479 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76144ddeb82b4e2d9f9ef217391d9d23" candidate_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4712fa936540f5b0728b87379eca37" is_pre_election: true
I20260929 16:01:15.605962 7855 raft_consensus.cc:2454] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb59afc1f1bf44bea2e2cfc3b651b845 in term 0.
I20260929 16:01:15.605998 7989 raft_consensus.cc:2454] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb59afc1f1bf44bea2e2cfc3b651b845 in term 0.
I20260929 16:01:15.606660 7655 leader_election.cc:302] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a49823299094d45a46d74b35452764c, fb59afc1f1bf44bea2e2cfc3b651b845, fc4712fa936540f5b0728b87379eca37; no voters:
I20260929 16:01:15.607123 8059 raft_consensus.cc:2781] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:01:15.607288 8059 raft_consensus.cc:482] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:01:15.607424 8059 raft_consensus.cc:3037] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:15.609625 8059 raft_consensus.cc:504] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:15.610301 8059 leader_election.cc:288] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [CANDIDATE]: Term 1 election: Requested vote from peers 1a49823299094d45a46d74b35452764c (127.23.222.67:33435), fc4712fa936540f5b0728b87379eca37 (127.23.222.66:40511)
I20260929 16:01:15.610891 7989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76144ddeb82b4e2d9f9ef217391d9d23" candidate_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a49823299094d45a46d74b35452764c"
I20260929 16:01:15.610929 7855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76144ddeb82b4e2d9f9ef217391d9d23" candidate_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4712fa936540f5b0728b87379eca37"
I20260929 16:01:15.611133 7989 raft_consensus.cc:3037] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:15.611133 7855 raft_consensus.cc:3037] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:15.614346 7855 raft_consensus.cc:2454] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb59afc1f1bf44bea2e2cfc3b651b845 in term 1.
I20260929 16:01:15.614341 7989 raft_consensus.cc:2454] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb59afc1f1bf44bea2e2cfc3b651b845 in term 1.
I20260929 16:01:15.614846 7654 leader_election.cc:302] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a49823299094d45a46d74b35452764c, fb59afc1f1bf44bea2e2cfc3b651b845; no voters:
I20260929 16:01:15.615330 8059 raft_consensus.cc:2781] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:15.615842 8059 raft_consensus.cc:686] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 LEADER]: Becoming Leader. State: Replica: fb59afc1f1bf44bea2e2cfc3b651b845, State: Running, Role: LEADER
I20260929 16:01:15.616315 8059 consensus_queue.cc:237] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:15.621019 7574 catalog_manager.cc:5495] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb59afc1f1bf44bea2e2cfc3b651b845 (127.23.222.65). New cstate: current_term: 1 leader_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } health_report { overall_health: UNKNOWN } } }
I20260929 16:01:15.686058 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.680356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.23.222.65:42973)
0929 16:01:15.680938 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.680943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.680962 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:01:15.681061 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.681065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.681161 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.681326 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:01:15.681733 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.682476 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.684055 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.684462 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.684563 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.685026 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:01:15.685533 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:01:15.685643 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.685746 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.685815 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":294,"threads_started":1}
I20260929 16:01:15.693985 7989 raft_consensus.cc:1261] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [term 1 FOLLOWER]: Refusing update from remote peer fb59afc1f1bf44bea2e2cfc3b651b845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:15.693990 7855 raft_consensus.cc:1261] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [term 1 FOLLOWER]: Refusing update from remote peer fb59afc1f1bf44bea2e2cfc3b651b845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:15.694996 8064 consensus_queue.cc:1035] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:15.695465 8059 consensus_queue.cc:1035] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:15.722820 7574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:15.723119 7574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:15.726630 7574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Sending DeleteTablet for 3 replicas of tablet 76144ddeb82b4e2d9f9ef217391d9d23
I20260929 16:01:15.727927 7700 tablet_service.cc:1514] Processing DeleteTablet for tablet 76144ddeb82b4e2d9f9ef217391d9d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:15 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:52564
I20260929 16:01:15.727959 7835 tablet_service.cc:1514] Processing DeleteTablet for tablet 76144ddeb82b4e2d9f9ef217391d9d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:15 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:34234
I20260929 16:01:15.728220 7969 tablet_service.cc:1514] Processing DeleteTablet for tablet 76144ddeb82b4e2d9f9ef217391d9d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:15 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:60916
I20260929 16:01:15.729049 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 7637
I20260929 16:01:15.745666 8086 tablet_replica.cc:330] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: stopping tablet replica
I20260929 16:01:15.746456 8086 raft_consensus.cc:2229] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:01:15.746515 8087 tablet_replica.cc:330] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: stopping tablet replica
I20260929 16:01:15.746940 8087 raft_consensus.cc:2229] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:15.747633 8087 raft_consensus.cc:2258] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:15.747601 8086 raft_consensus.cc:2258] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:15.750154 8086 ts_tablet_manager.cc:1905] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:15.751327 8087 ts_tablet_manager.cc:1905] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:15.756927 8085 tablet_replica.cc:330] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: stopping tablet replica
I20260929 16:01:15.758495 8085 raft_consensus.cc:2229] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:15.759094 8085 raft_consensus.cc:2258] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 16:01:15 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 16:01:15.761814 8085 ts_tablet_manager.cc:1905] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:15.762391 8087 ts_tablet_manager.cc:1918] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:15.762633 8087 log.cc:1196] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/76144ddeb82b4e2d9f9ef217391d9d23
I20260929 16:01:15.765662 8086 ts_tablet_manager.cc:1918] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:15.766000 8087 ts_tablet_manager.cc:1939] T 76144ddeb82b4e2d9f9ef217391d9d23 P 1a49823299094d45a46d74b35452764c: Deleting consensus metadata
I20260929 16:01:15.768211 7560 catalog_manager.cc:4841] TS 1a49823299094d45a46d74b35452764c (127.23.222.67:33435): tablet 76144ddeb82b4e2d9f9ef217391d9d23 (table test-table [id=2d77d4a987da4392abc73a5c5d9ee5d7]) successfully deleted
I20260929 16:01:15.768836 8086 log.cc:1196] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/76144ddeb82b4e2d9f9ef217391d9d23
I20260929 16:01:15.769562 8086 ts_tablet_manager.cc:1939] T 76144ddeb82b4e2d9f9ef217391d9d23 P fb59afc1f1bf44bea2e2cfc3b651b845: Deleting consensus metadata
I20260929 16:01:15.770140 8085 ts_tablet_manager.cc:1918] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:15.770414 8085 log.cc:1196] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/76144ddeb82b4e2d9f9ef217391d9d23
I20260929 16:01:15.771075 8085 ts_tablet_manager.cc:1939] T 76144ddeb82b4e2d9f9ef217391d9d23 P fc4712fa936540f5b0728b87379eca37: Deleting consensus metadata
I20260929 16:01:15.771106 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.731779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.23.222.65:42973)
0929 16:01:15.738416 (+ 6637us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.738674 (+ 258us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.738702 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:01:15.738781 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.738789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.738889 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.739047 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:15.742200 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.743470 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.748139 (+ 4669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.748814 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.748918 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.761547 (+ 12629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:15.761570 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:15.761588 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:15.761652 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:15.766434 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.767669 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.767682 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.767686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.767917 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.768521 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.768528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.768531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.769043 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:15.769404 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.770187 (+ 783us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.770528 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6459}
I20260929 16:01:15.771562 7560 catalog_manager.cc:4841] TS fb59afc1f1bf44bea2e2cfc3b651b845 (127.23.222.65:42973): tablet 76144ddeb82b4e2d9f9ef217391d9d23 (table test-table [id=2d77d4a987da4392abc73a5c5d9ee5d7]) successfully deleted
I20260929 16:01:15.772648 7760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:15.773663 7561 catalog_manager.cc:4841] TS fc4712fa936540f5b0728b87379eca37 (127.23.222.66:40511): tablet 76144ddeb82b4e2d9f9ef217391d9d23 (table test-table [id=2d77d4a987da4392abc73a5c5d9ee5d7]) successfully deleted
I20260929 16:01:15.873174 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 7637
I20260929 16:01:15.893165 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 7772
Sep 29 16:01:15 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 16:01:15.909237 8051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:15.893957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.23.222.66:40511)
0929 16:01:15.894151 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:15.894158 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:15.894253 (+ 95us) server_negotiation.cc:407] Connection header received
0929 16:01:15.894466 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:15.894472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:15.894561 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:15.894670 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:15.896088 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.896852 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.898176 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:15.898613 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:15.898752 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:15.903361 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:15.903377 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:15.903393 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:15.903431 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:15.906153 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.906912 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.906920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.906922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.907064 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:15.907620 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:15.907627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:15.907630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:15.908028 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:15.908345 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:15.908663 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 16:01:15.908887 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:01:15.909914 7895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:15.998930 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 7772
I20260929 16:01:16.016494 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 7906
Sep 29 16:01:16 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 16:01:16.032938 8052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:16.017339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.23.222.67:33435)
0929 16:01:16.017546 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:16.017552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:16.017573 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:01:16.017732 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:16.017737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:16.017816 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:16.017924 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:16.019428 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.020212 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.021539 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:16.021957 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:16.022104 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:16.026795 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:16.026812 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:16.026828 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:16.026871 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:16.029704 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:16.030513 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:16.030521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:16.030524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:16.030675 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:16.031098 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:16.031104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:16.031107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:16.031674 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:16.031999 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:16.032321 (+ 322us) server_negotiation.cc:299] Negotiation successful
0929 16:01:16.032575 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:01:16.033723 8029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:16.122668 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 7906
I20260929 16:01:16.139406 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 7543
I20260929 16:01:16.140561 7605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:16.255787 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 7543
I20260929 16:01:36.278039 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37599
--webserver_interface=127.23.222.126
--webserver_port=42817
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:37599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:36.608901 8093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:36.609277 8093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:36.609457 8093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:36.616952 8093 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:01:36.617116 8093 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:01:36.617211 8093 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:01:36.617293 8093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:36.617394 8093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:36.617502 8093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:36.617592 8093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:36.617676 8093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:36.627694 8093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:37599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=42817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:01:36.629122 8093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:36.630497 8093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:36.654345 8103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:36.654320 8100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:36.654389 8101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:36.655723 8093 server_base.cc:1034] running on GCE node
I20260929 16:01:36.656605 8093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:36.658192 8093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:36.659415 8093 hybrid_clock.cc:648] HybridClock initialized: now 1790697696659354 us; error 37 us; skew 500 ppm
I20260929 16:01:36.659770 8093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:36 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697696, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:36.669516 8093 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:01:36.671454 8093 webserver.cc:460] Webserver started at http://127.23.222.126:42817/ using document root <none> and password file <none>
I20260929 16:01:36.672297 8093 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:36.672438 8093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:36.678767 8093 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 16:01:36.682005 8110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:36.683791 8093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:01:36.684005 8093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "73588ebc21114361a4f6fe399cc48cdd"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "7f296754d45deaceb3c5ca4a2f093d7e"
server_key_iv: "bc6afcbf64f3ae71852dfbedd5a9bcb2"
server_key_version: "encryptionkey@0"
I20260929 16:01:36.685164 8093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:36.699949 8093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:36.704417 8093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:36.704766 8093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:36.725678 8093 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:37599
I20260929 16:01:36.725703 8161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:37599 every 8 connection(s)
I20260929 16:01:36.727439 8093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:01:36.731771 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 8093
I20260929 16:01:36.732560 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:42973
--local_ip_for_outbound_sockets=127.23.222.65
--tserver_master_addrs=127.23.222.126:37599
--webserver_port=32789
--webserver_interface=127.23.222.65
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:01:36.734918 8162 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:01:36.739647 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Bootstrap starting.
I20260929 16:01:36.759948 8162 log.cc:824] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:36.772871 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:01:36.773494 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Bootstrap complete.
I20260929 16:01:36.782958 8162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:01:36.783854 8162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73588ebc21114361a4f6fe399cc48cdd, State: Initialized, Role: FOLLOWER
I20260929 16:01:36.784623 8162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:01:36.784875 8162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:01:36.785009 8162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:01:36.785207 8162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 1 FOLLOWER]: Advancing to term 2
I20260929 16:01:36.789345 8162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:01:36.789918 8162 leader_election.cc:302] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73588ebc21114361a4f6fe399cc48cdd; no voters:
I20260929 16:01:36.790633 8162 leader_election.cc:288] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 16:01:36.790774 8166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 16:01:36.793308 8162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:01:36.795616 8166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 2 LEADER]: Becoming Leader. State: Replica: 73588ebc21114361a4f6fe399cc48cdd, State: Running, Role: LEADER
I20260929 16:01:36.796519 8166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:01:36.799518 8167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "73588ebc21114361a4f6fe399cc48cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } } }
I20260929 16:01:36.799916 8167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:36.799623 8168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73588ebc21114361a4f6fe399cc48cdd. Latest consensus state: current_term: 2 leader_uuid: "73588ebc21114361a4f6fe399cc48cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } } }
I20260929 16:01:36.800047 8168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: This master's current role is: LEADER
I20260929 16:01:36.804829 8178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:01:36.814844 8178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:01:36.816874 8178 catalog_manager.cc:1259] Loaded cluster ID: d6e40febac324fffa2e863424f1506b8
I20260929 16:01:36.817075 8178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:01:36.822378 8178 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:01:36.824585 8178 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Loaded TSK: 0
I20260929 16:01:36.826911 8184 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:01:36.850772 8178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Generated new TSK 1
I20260929 16:01:36.851219 8178 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 16:01:37.099282 8164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:37.099602 8164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:37.099761 8164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:37.107046 8164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:37.107208 8164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:37.107404 8164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:01:37.117532 8164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:42973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=32789
--tserver_master_addrs=127.23.222.126:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:01:37.118860 8164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:37.120288 8164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:37.135397 8191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:37.135417 8190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:37.136351 8193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:37.136497 8164 server_base.cc:1034] running on GCE node
I20260929 16:01:37.137202 8164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:37.138190 8164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:37.139417 8164 hybrid_clock.cc:648] HybridClock initialized: now 1790697697139353 us; error 57 us; skew 500 ppm
I20260929 16:01:37.139747 8164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:37 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:37.143081 8164 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:01:37.144625 8164 webserver.cc:460] Webserver started at http://127.23.222.65:32789/ using document root <none> and password file <none>
I20260929 16:01:37.145370 8164 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:37.145514 8164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:37.151175 8164 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 16:01:37.153992 8200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:37.155568 8164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:01:37.155766 8164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "fb59afc1f1bf44bea2e2cfc3b651b845"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "22b536b1cc08478f648d03c391d33604"
server_key_iv: "5774bca8c059842e9eebe0b50dac0be0"
server_key_version: "encryptionkey@0"
I20260929 16:01:37.156821 8164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:37.183596 8164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:37.188441 8164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:37.188823 8164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:37.189986 8164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:37.191633 8164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:37.191776 8164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:37.191920 8164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:37.192028 8164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:37.227814 8164 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:42973
I20260929 16:01:37.227907 8313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:42973 every 8 connection(s)
I20260929 16:01:37.229332 8164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:01:37.237388 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 8164
I20260929 16:01:37.238034 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:40511
--local_ip_for_outbound_sockets=127.23.222.66
--tserver_master_addrs=127.23.222.126:37599
--webserver_port=42911
--webserver_interface=127.23.222.66
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:01:37 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:01:37.253291 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:37.232451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:40355 (local address 127.23.222.126:37599)
0929 16:01:37.233119 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:37.233139 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:37.233661 (+ 522us) server_negotiation.cc:407] Connection header received
0929 16:01:37.235891 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:37.235904 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:37.236225 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:37.236590 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:37.238186 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.239554 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.241129 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.241794 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.241996 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:37.245519 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:37.245527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:37.245542 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:37.245577 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:37.248758 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:37.249506 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:37.249514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:37.249519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:37.249684 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:37.250120 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:37.250126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:37.250129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:37.250588 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:37.250797 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:37.251644 (+ 847us) server_negotiation.cc:299] Negotiation successful
0929 16:01:37.252045 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":263,"threads_started":1}
I20260929 16:01:37.253295 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:37.232648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:40355)
0929 16:01:37.233476 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:37.233518 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:37.235343 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:37.236737 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:37.236751 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:37.237391 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:37.237989 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:37.238006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.239725 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.239732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:37.240972 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:37.240984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.242032 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.242040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:37.242320 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:37.243009 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:37.243032 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:37.245236 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:37.248898 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:37.248907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:37.248922 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:37.249359 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:37.249802 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:37.249807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:37.249811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:37.250003 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:37.250751 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:37.250758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:37.251080 (+ 322us) client_negotiation.cc:769] Sending connection context
0929 16:01:37.251416 (+ 336us) client_negotiation.cc:240] Negotiation successful
0929 16:01:37.251960 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":338,"threads_started":1}
I20260929 16:01:37.256484 8314 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37599
I20260929 16:01:37.256870 8314 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:37.257834 8314 heartbeater.cc:511] Master 127.23.222.126:37599 requested a full tablet report, sending...
I20260929 16:01:37.260558 8127 ts_manager.cc:194] Registered new tserver with Master: fb59afc1f1bf44bea2e2cfc3b651b845 (127.23.222.65:42973)
I20260929 16:01:37.262974 8127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:40355
W20260929 16:01:37.575686 8318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:37.575973 8318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:37.576068 8318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:37.583287 8318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:37.583413 8318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:37.583534 8318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:01:37.593567 8318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=42911
--tserver_master_addrs=127.23.222.126:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:01:37.594903 8318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:37.596251 8318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:37.609139 8323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:37.609139 8324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:37.609169 8326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:37.610101 8318 server_base.cc:1034] running on GCE node
I20260929 16:01:37.610750 8318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:37.611723 8318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:37.612955 8318 hybrid_clock.cc:648] HybridClock initialized: now 1790697697612907 us; error 58 us; skew 500 ppm
I20260929 16:01:37.613339 8318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:37 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:37.616394 8318 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:01:37.617897 8318 webserver.cc:460] Webserver started at http://127.23.222.66:42911/ using document root <none> and password file <none>
I20260929 16:01:37.618587 8318 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:37.618752 8318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:37.624293 8318 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 16:01:37.626972 8333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:37.628384 8318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 16:01:37.628580 8318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "fc4712fa936540f5b0728b87379eca37"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "42aeef3c5ce0e819beb3a2b84423a9c0"
server_key_iv: "1196a270605030c061dbd36450a35526"
server_key_version: "encryptionkey@0"
I20260929 16:01:37.629593 8318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:37.663125 8318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:37.667807 8318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:37.668155 8318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:37.669425 8318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:37.671015 8318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:37.671147 8318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:37.671314 8318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:37.671411 8318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:37.707103 8318 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:40511
I20260929 16:01:37.707151 8446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:40511 every 8 connection(s)
I20260929 16:01:37.708762 8318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:01:37.711445 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 8318
I20260929 16:01:37.712178 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:33435
--local_ip_for_outbound_sockets=127.23.222.67
--tserver_master_addrs=127.23.222.126:37599
--webserver_port=33269
--webserver_interface=127.23.222.67
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 16:01:37 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697697, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:01:37.732726 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:37.711416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:45567 (local address 127.23.222.126:37599)
0929 16:01:37.711628 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:37.711633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:37.713091 (+ 1458us) server_negotiation.cc:407] Connection header received
0929 16:01:37.715886 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:37.715895 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:37.716033 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:37.716142 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:37.718397 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.719558 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.721153 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.721595 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.721748 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:37.726037 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:37.726042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:37.726046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:37.726106 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:37.729453 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:37.730198 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:37.730206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:37.730211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:37.730344 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:37.730796 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:37.730802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:37.730806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:37.731168 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:37.731378 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:37.732095 (+ 717us) server_negotiation.cc:299] Negotiation successful
0929 16:01:37.732362 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:01:37.734273 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:37.712149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:45567)
0929 16:01:37.712936 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:37.712973 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:37.715468 (+ 2495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:37.716629 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:37.716642 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:37.717534 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:37.718192 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:37.718213 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.719736 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.719740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:37.720997 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:37.721009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:37.721844 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:37.721851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:37.722199 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:37.722908 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:37.722943 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:37.725770 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:37.729617 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:37.729627 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:37.729643 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:37.730069 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:37.730462 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:37.730467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:37.730470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:37.730678 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:37.731340 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:37.731349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:37.731705 (+ 356us) client_negotiation.cc:769] Sending connection context
0929 16:01:37.731986 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 16:01:37.732609 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":321,"threads_started":1}
I20260929 16:01:37.735872 8447 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37599
I20260929 16:01:37.736212 8447 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:37.736970 8447 heartbeater.cc:511] Master 127.23.222.126:37599 requested a full tablet report, sending...
I20260929 16:01:37.738512 8127 ts_manager.cc:194] Registered new tserver with Master: fc4712fa936540f5b0728b87379eca37 (127.23.222.66:40511)
I20260929 16:01:37.739491 8127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:45567
W20260929 16:01:38.052635 8450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:38.052938 8450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:38.053099 8450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:38.060621 8450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:38.060784 8450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:38.060916 8450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:01:38.071213 8450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:33435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=33269
--tserver_master_addrs=127.23.222.126:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:01:38.072536 8450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:38.073900 8450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:38.089327 8455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:38.089398 8458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:38.089370 8456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:38.090372 8450 server_base.cc:1034] running on GCE node
I20260929 16:01:38.090953 8450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:38.091969 8450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:38.093178 8450 hybrid_clock.cc:648] HybridClock initialized: now 1790697698093137 us; error 37 us; skew 500 ppm
I20260929 16:01:38.093544 8450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:38 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697698, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:38.096913 8450 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:01:38.098490 8450 webserver.cc:460] Webserver started at http://127.23.222.67:33269/ using document root <none> and password file <none>
I20260929 16:01:38.099223 8450 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:38.099438 8450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:38.104947 8450 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 16:01:38.107841 8465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:38.109251 8450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:01:38.109440 8450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "1a49823299094d45a46d74b35452764c"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "9d2f009bb6f70fc941b52c8933ce3a18"
server_key_iv: "313197c773242466086ea7c0b5063580"
server_key_version: "encryptionkey@0"
I20260929 16:01:38.110464 8450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:38.153442 8450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:38.157840 8450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:38.158147 8450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:38.159195 8450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:01:38.160848 8450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:01:38.160982 8450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:38.161114 8450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:01:38.161213 8450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:38.201269 8450 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:33435
I20260929 16:01:38.201370 8578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:33435 every 8 connection(s)
I20260929 16:01:38.202939 8450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:01:38.213047 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 8450
Sep 29 16:01:38 dist-test-slave-cq88 krb5kdc[7530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697698, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:01:38.226325 8316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.205936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:50463 (local address 127.23.222.126:37599)
0929 16:01:38.206176 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.206180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.207418 (+ 1238us) server_negotiation.cc:407] Connection header received
0929 16:01:38.209309 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.209314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.209388 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.209488 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:38.211580 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.212493 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.215269 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.215822 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.215972 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.219733 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:38.219738 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:38.219741 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:38.219785 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:38.222810 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.223785 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.223790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.223792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.223912 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.224412 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.224415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.224417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.224709 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:38.224885 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.225636 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.225944 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 16:01:38.227710 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.206387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:50463)
0929 16:01:38.207194 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.207253 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.209002 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.209868 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.209881 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.210668 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:01:38.211353 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.211374 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.212738 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.212745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.215075 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.215097 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.216086 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.216094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.216372 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.217118 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:01:38.217142 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:01:38.219489 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:01:38.222995 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:38.223005 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:38.223026 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:38.223638 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:38.224048 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:01:38.224053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:01:38.224057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:01:38.224270 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:01:38.224857 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:01:38.224866 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:01:38.225256 (+ 390us) client_negotiation.cc:769] Sending connection context
0929 16:01:38.225536 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 16:01:38.226142 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":293,"threads_started":1}
I20260929 16:01:38.228957 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.214179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.23.222.126:37599)
0929 16:01:38.215140 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.215147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.215179 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:01:38.215269 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.215276 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.215345 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.215473 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:38.217191 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.217930 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.219553 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.220056 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.220244 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.223024 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:38.223030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:38.223034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:38.223067 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:38.225858 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.226667 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.226673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.226677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.226808 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.227290 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.227297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.227299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.227648 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:38.227935 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.228293 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.228536 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":692,"threads_started":1}
I20260929 16:01:38.229525 8579 heartbeater.cc:347] Connected to a master server at 127.23.222.126:37599
I20260929 16:01:38.229847 8579 heartbeater.cc:464] Registering TS with master...
I20260929 16:01:38.230769 8579 heartbeater.cc:511] Master 127.23.222.126:37599 requested a full tablet report, sending...
I20260929 16:01:38.232370 8127 ts_manager.cc:194] Registered new tserver with Master: 1a49823299094d45a46d74b35452764c (127.23.222.67:33435)
I20260929 16:01:38.233310 8127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:50463
I20260929 16:01:38.244627 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:01:38.251461 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.245727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.23.222.126:37599)
0929 16:01:38.245905 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.245910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.246397 (+ 487us) server_negotiation.cc:407] Connection header received
0929 16:01:38.246610 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.246614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.246773 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.246913 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:01:38.247377 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.248108 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.249526 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.249959 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.250060 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.250591 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:01:38.250852 (+ 261us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 16:01:38.251165 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38220: token expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 16:01:38.251624 8583 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38220: token expired
W20260929 16:01:38.252077 8039 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.222.126:37599)
I20260929 16:01:38.265461 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.252534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.23.222.126:37599)
0929 16:01:38.252687 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.252690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.252705 (+ 15us) server_negotiation.cc:407] Connection header received
0929 16:01:38.252827 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.252830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.252888 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.252964 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:38.254294 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.255041 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.256507 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.256912 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.257009 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.259899 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:38.259903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:38.259907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:38.259942 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:38.262793 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.263654 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.263666 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.263669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.263812 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.264179 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.264182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.264184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.264519 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:38.264711 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.264995 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.265206 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:01:38.266700 8314 heartbeater.cc:503] Master 127.23.222.126:37599 was elected leader, sending a full tablet report...
I20260929 16:01:38.269260 8126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:01:38.277431 8126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:01:38.304267 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.290248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:42973 (local address 127.0.0.1:39516)
0929 16:01:38.291143 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.291192 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.291399 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.292643 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.292654 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.292711 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:38.293260 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.293277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.296004 (+ 2727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.296013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.299486 (+ 3473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.299515 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.302776 (+ 3261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.302786 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.303124 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.303201 (+ 77us) client_negotiation.cc:769] Sending connection context
0929 16:01:38.303388 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 16:01:38.303718 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":977,"threads_started":1}
I20260929 16:01:38.305796 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.290583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.23.222.65:42973)
0929 16:01:38.291291 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.291299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.291332 (+ 33us) server_negotiation.cc:407] Connection header received
0929 16:01:38.291738 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.291748 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.292060 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.292301 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:38.293536 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.295772 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.299794 (+ 4022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.301999 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.302187 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.302535 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.305044 (+ 2509us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.305316 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":322,"threads_started":1}
I20260929 16:01:38.313757 8249 tablet_service.cc:1467] Processing CreateTablet for tablet f26db5ba08374b8bbf230664dc57f919 (DEFAULT_TABLE table=test-table [id=440e17a75ef548e1a2b2cae18c83c445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:38.315835 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26db5ba08374b8bbf230664dc57f919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:38.316443 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.291661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:40511 (local address 127.0.0.1:40286)
0929 16:01:38.304631 (+ 12970us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.304668 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.304809 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.307496 (+ 2687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.307505 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.307548 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:38.307852 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.307867 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.310650 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.310671 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.313332 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.313375 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.315560 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.315566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.315766 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.315804 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 16:01:38.315906 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 16:01:38.316071 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12823,"mutex_wait_us":29,"thread_start_us":21344,"threads_started":1}
I20260929 16:01:38.316816 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.291646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.23.222.66:40511)
0929 16:01:38.306566 (+ 14920us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.306581 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.306623 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:01:38.306764 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.306777 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.307146 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.308095 (+ 949us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:38.308426 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.310443 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.313584 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.315363 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.315514 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.315715 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.316103 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.316295 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14689,"thread_start_us":14555,"threads_started":1}
I20260929 16:01:38.317800 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.292155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:33435 (local address 127.0.0.1:50704)
0929 16:01:38.308312 (+ 16157us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.308331 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.308453 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.309541 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.309546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.309571 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:38.309946 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.309959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.312073 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.312090 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.314686 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.314700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.316996 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.317002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.317152 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.317176 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:01:38.317281 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:01:38.317416 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16026,"mutex_wait_us":1,"thread_start_us":16816,"threads_started":1}
I20260929 16:01:38.317837 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.292910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.23.222.67:33435)
0929 16:01:38.307822 (+ 14912us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.307833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.308587 (+ 754us) server_negotiation.cc:407] Connection header received
0929 16:01:38.308714 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.308722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.308992 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.309209 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:38.310175 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.311901 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.314909 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.316423 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.316561 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.316826 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.317340 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.317503 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14723,"thread_start_us":15876,"threads_started":1}
I20260929 16:01:38.320825 8381 tablet_service.cc:1467] Processing CreateTablet for tablet f26db5ba08374b8bbf230664dc57f919 (DEFAULT_TABLE table=test-table [id=440e17a75ef548e1a2b2cae18c83c445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:38.321090 8513 tablet_service.cc:1467] Processing CreateTablet for tablet f26db5ba08374b8bbf230664dc57f919 (DEFAULT_TABLE table=test-table [id=440e17a75ef548e1a2b2cae18c83c445]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:01:38.322425 8381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26db5ba08374b8bbf230664dc57f919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:38.322515 8513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26db5ba08374b8bbf230664dc57f919. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:01:38.332720 8595 tablet_bootstrap.cc:492] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: Bootstrap starting.
I20260929 16:01:38.336068 8596 tablet_bootstrap.cc:492] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: Bootstrap starting.
I20260929 16:01:38.337281 8595 tablet_bootstrap.cc:654] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:38.338606 8595 log.cc:824] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:38.338891 8597 tablet_bootstrap.cc:492] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: Bootstrap starting.
I20260929 16:01:38.339988 8596 tablet_bootstrap.cc:654] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:38.341331 8596 log.cc:824] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:38.342731 8597 tablet_bootstrap.cc:654] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: Neither blocks nor log segments found. Creating new log.
I20260929 16:01:38.343758 8595 tablet_bootstrap.cc:492] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: No bootstrap required, opened a new log
I20260929 16:01:38.344117 8595 ts_tablet_manager.cc:1397] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260929 16:01:38.344245 8597 log.cc:824] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: Log is configured to *not* fsync() on all Append() calls
I20260929 16:01:38.344937 8596 tablet_bootstrap.cc:492] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: No bootstrap required, opened a new log
I20260929 16:01:38.345249 8596 ts_tablet_manager.cc:1397] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 16:01:38.348080 8597 tablet_bootstrap.cc:492] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: No bootstrap required, opened a new log
I20260929 16:01:38.348407 8597 ts_tablet_manager.cc:1397] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260929 16:01:38.352919 8596 raft_consensus.cc:348] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.353372 8596 raft_consensus.cc:374] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:38.353471 8596 raft_consensus.cc:729] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a49823299094d45a46d74b35452764c, State: Initialized, Role: FOLLOWER
I20260929 16:01:38.354189 8596 consensus_queue.cc:260] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.355471 8579 heartbeater.cc:503] Master 127.23.222.126:37599 was elected leader, sending a full tablet report...
I20260929 16:01:38.355743 8595 raft_consensus.cc:348] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.356248 8595 raft_consensus.cc:374] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:38.356348 8595 raft_consensus.cc:729] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb59afc1f1bf44bea2e2cfc3b651b845, State: Initialized, Role: FOLLOWER
I20260929 16:01:38.357048 8595 consensus_queue.cc:260] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.357806 8596 ts_tablet_manager.cc:1428] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: Time spent starting tablet: real 0.012s user 0.004s sys 0.008s
I20260929 16:01:38.360455 8597 raft_consensus.cc:348] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.361042 8597 raft_consensus.cc:374] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:01:38.361151 8597 raft_consensus.cc:729] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4712fa936540f5b0728b87379eca37, State: Initialized, Role: FOLLOWER
I20260929 16:01:38.361832 8595 ts_tablet_manager.cc:1428] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260929 16:01:38.361940 8597 consensus_queue.cc:260] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.363646 8447 heartbeater.cc:503] Master 127.23.222.126:37599 was elected leader, sending a full tablet report...
I20260929 16:01:38.364483 8597 ts_tablet_manager.cc:1428] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260929 16:01:38.454835 8580 tablet.cc:2369] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:01:38.460884 8448 tablet.cc:2369] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:01:38.481869 8315 tablet.cc:2369] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:01:38.713188 8601 raft_consensus.cc:482] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:38.713512 8601 raft_consensus.cc:504] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.715492 8601 leader_election.cc:288] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb59afc1f1bf44bea2e2cfc3b651b845 (127.23.222.65:42973), fc4712fa936540f5b0728b87379eca37 (127.23.222.66:40511)
I20260929 16:01:38.719624 8602 raft_consensus.cc:482] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:01:38.719951 8602 raft_consensus.cc:504] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.722056 8602 leader_election.cc:288] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a49823299094d45a46d74b35452764c (127.23.222.67:33435), fc4712fa936540f5b0728b87379eca37 (127.23.222.66:40511)
I20260929 16:01:38.723980 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.716079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:39121 (local address 127.23.222.66:40511)
0929 16:01:38.716654 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.716662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.716688 (+ 26us) server_negotiation.cc:407] Connection header received
0929 16:01:38.716800 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.716806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.716862 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.716967 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:38.717571 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.718634 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.721504 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.722744 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.722904 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.722983 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.723591 (+ 608us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.723718 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428}
I20260929 16:01:38.724094 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.715919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:40511 (local address 127.23.222.67:39121)
0929 16:01:38.716529 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.716548 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.716680 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.716981 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.716986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.717009 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:38.717397 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.717408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.718817 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.718826 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.721330 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.721346 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.723222 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.723226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.723422 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.723446 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 16:01:38.723551 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:01:38.723696 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":281,"thread_start_us":349,"threads_started":1}
I20260929 16:01:38.725463 8402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26db5ba08374b8bbf230664dc57f919" candidate_uuid: "1a49823299094d45a46d74b35452764c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4712fa936540f5b0728b87379eca37" is_pre_election: true
I20260929 16:01:38.726233 8402 raft_consensus.cc:2454] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a49823299094d45a46d74b35452764c in term 0.
I20260929 16:01:38.727248 8468 leader_election.cc:302] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a49823299094d45a46d74b35452764c, fc4712fa936540f5b0728b87379eca37; no voters:
I20260929 16:01:38.727720 8601 raft_consensus.cc:2781] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:01:38.727859 8601 raft_consensus.cc:482] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:01:38.727974 8601 raft_consensus.cc:3037] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:38.731333 8601 raft_consensus.cc:504] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.732517 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.715659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:42973 (local address 127.23.222.67:33131)
0929 16:01:38.716077 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.716097 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.716240 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.717151 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.717156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.717196 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:38.717451 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.717462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.718895 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.718901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.721939 (+ 3038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.721955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.731598 (+ 9643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.731606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.731877 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.731922 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:01:38.731999 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 16:01:38.732181 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":83}
I20260929 16:01:38.732734 8402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26db5ba08374b8bbf230664dc57f919" candidate_uuid: "1a49823299094d45a46d74b35452764c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4712fa936540f5b0728b87379eca37"
I20260929 16:01:38.732939 8402 raft_consensus.cc:3037] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:38.733631 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.716572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:33131 (local address 127.23.222.65:42973)
0929 16:01:38.716766 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.716772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.716789 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:01:38.716855 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.716860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.716924 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.717001 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:38.717651 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.718726 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.722146 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.725230 (+ 3084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.733022 (+ 7792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.733110 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.733180 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.733339 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:01:38.736286 8269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26db5ba08374b8bbf230664dc57f919" candidate_uuid: "1a49823299094d45a46d74b35452764c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" is_pre_election: true
I20260929 16:01:38.736554 8402 raft_consensus.cc:2454] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a49823299094d45a46d74b35452764c in term 1.
I20260929 16:01:38.736796 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.723202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:52139 (local address 127.23.222.66:40511)
0929 16:01:38.724226 (+ 1024us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.724233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.728726 (+ 4493us) server_negotiation.cc:407] Connection header received
0929 16:01:38.728960 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.728965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.729032 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.729099 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:38.729868 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.730959 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.734100 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.735504 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.735657 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.735724 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.736338 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.736483 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":4982,"threads_started":1}
I20260929 16:01:38.736773 8269 raft_consensus.cc:2454] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a49823299094d45a46d74b35452764c in term 0.
I20260929 16:01:38.736986 8468 leader_election.cc:302] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a49823299094d45a46d74b35452764c, fc4712fa936540f5b0728b87379eca37; no voters:
I20260929 16:01:38.737555 8601 leader_election.cc:288] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [CANDIDATE]: Term 1 election: Requested vote from peers fb59afc1f1bf44bea2e2cfc3b651b845 (127.23.222.65:42973), fc4712fa936540f5b0728b87379eca37 (127.23.222.66:40511)
I20260929 16:01:38.737391 8268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26db5ba08374b8bbf230664dc57f919" candidate_uuid: "1a49823299094d45a46d74b35452764c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845"
I20260929 16:01:38.737716 8268 raft_consensus.cc:3037] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:01:38.737797 8601 raft_consensus.cc:2781] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:01:38.737952 8601 raft_consensus.cc:686] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 1 LEADER]: Becoming Leader. State: Replica: 1a49823299094d45a46d74b35452764c, State: Running, Role: LEADER
I20260929 16:01:38.738636 8601 consensus_queue.cc:237] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } }
I20260929 16:01:38.741058 8268 raft_consensus.cc:2454] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a49823299094d45a46d74b35452764c in term 1.
I20260929 16:01:38.741884 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.722593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:41739 (local address 127.23.222.67:33435)
0929 16:01:38.723112 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.723117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.725268 (+ 2151us) server_negotiation.cc:407] Connection header received
0929 16:01:38.725517 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.725522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.725594 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.725649 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:01:38.726402 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.727461 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.739283 (+ 11822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.740624 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.740765 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.740863 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.741483 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.741615 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381}
I20260929 16:01:38.742514 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.722223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:33435 (local address 127.23.222.65:41739)
0929 16:01:38.725168 (+ 2945us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.725190 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.725378 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.725841 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.725845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.725870 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:38.726258 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.726269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.727630 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.727635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.739040 (+ 11405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.739054 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.741095 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.741102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.741286 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.741320 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 16:01:38.742023 (+ 703us) client_negotiation.cc:240] Negotiation successful
0929 16:01:38.742165 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2810,"thread_start_us":5465,"threads_started":1}
I20260929 16:01:38.743510 8534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26db5ba08374b8bbf230664dc57f919" candidate_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a49823299094d45a46d74b35452764c" is_pre_election: true
I20260929 16:01:38.745118 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.722649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:40511 (local address 127.23.222.65:52139)
0929 16:01:38.728631 (+ 5982us) negotiation.cc:107] Waiting for socket to connect
0929 16:01:38.728651 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:01:38.728825 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:01:38.729301 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:01:38.729307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:01:38.729333 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:01:38.729685 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.729695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.731606 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.731610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.733909 (+ 2299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:01:38.733925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.735956 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.735962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:01:38.736145 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.736175 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:01:38.736994 (+ 819us) client_negotiation.cc:240] Negotiation successful
0929 16:01:38.737148 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5843,"thread_start_us":11099,"threads_started":1}
I20260929 16:01:38.745988 8402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26db5ba08374b8bbf230664dc57f919" candidate_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc4712fa936540f5b0728b87379eca37" is_pre_election: true
I20260929 16:01:38.746327 8402 raft_consensus.cc:2379] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fb59afc1f1bf44bea2e2cfc3b651b845 in current term 1: Already voted for candidate 1a49823299094d45a46d74b35452764c in this term.
I20260929 16:01:38.746500 8127 catalog_manager.cc:5495] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a49823299094d45a46d74b35452764c (127.23.222.67). New cstate: current_term: 1 leader_uuid: "1a49823299094d45a46d74b35452764c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a49823299094d45a46d74b35452764c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 33435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 } health_report { overall_health: UNKNOWN } } }
I20260929 16:01:38.747115 8203 leader_election.cc:302] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb59afc1f1bf44bea2e2cfc3b651b845; no voters: 1a49823299094d45a46d74b35452764c, fc4712fa936540f5b0728b87379eca37
I20260929 16:01:38.747629 8602 raft_consensus.cc:2726] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 16:01:38.870456 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.864796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.23.222.67:33435)
0929 16:01:38.865081 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.865090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.865382 (+ 292us) server_negotiation.cc:407] Connection header received
0929 16:01:38.865560 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.865564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.865648 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.865778 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:01:38.866129 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.866849 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.868256 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.868654 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.868753 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.869227 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:01:38.869831 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:01:38.869955 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.870044 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.870112 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 16:01:38.878774 8402 raft_consensus.cc:1261] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 1 FOLLOWER]: Refusing update from remote peer 1a49823299094d45a46d74b35452764c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:38.878818 8268 raft_consensus.cc:1261] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 FOLLOWER]: Refusing update from remote peer 1a49823299094d45a46d74b35452764c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:01:38.879830 8608 consensus_queue.cc:1035] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc4712fa936540f5b0728b87379eca37" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:38.880203 8601 consensus_queue.cc:1035] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb59afc1f1bf44bea2e2cfc3b651b845" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 42973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:01:38.890123 8614 mvcc.cc:205] Tried to move back new op lower bound from 7334697774596468736 to 7334697774039478272. Current Snapshot: MvccSnapshot[applied={T|T < 7334697774596468736}]
I20260929 16:01:38.890472 8612 mvcc.cc:205] Tried to move back new op lower bound from 7334697774596468736 to 7334697774039478272. Current Snapshot: MvccSnapshot[applied={T|T < 7334697774596468736}]
I20260929 16:01:38.905268 8127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:38.905545 8127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:01:38.908824 8127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Sending DeleteTablet for 3 replicas of tablet f26db5ba08374b8bbf230664dc57f919
I20260929 16:01:38.910076 8513 tablet_service.cc:1514] Processing DeleteTablet for tablet f26db5ba08374b8bbf230664dc57f919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:38 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:50704
I20260929 16:01:38.910214 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 8164
I20260929 16:01:38.910293 8249 tablet_service.cc:1514] Processing DeleteTablet for tablet f26db5ba08374b8bbf230664dc57f919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:38 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:39516
I20260929 16:01:38.910985 8381 tablet_service.cc:1514] Processing DeleteTablet for tablet f26db5ba08374b8bbf230664dc57f919 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:01:38 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:40286
I20260929 16:01:38.934130 8626 tablet_replica.cc:330] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: stopping tablet replica
I20260929 16:01:38.935081 8626 raft_consensus.cc:2229] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:38.935971 8624 tablet_replica.cc:330] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: stopping tablet replica
I20260929 16:01:38.936738 8626 raft_consensus.cc:2258] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:38.937395 8624 raft_consensus.cc:2229] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:01:38.940320 8626 ts_tablet_manager.cc:1905] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:38.941018 8624 raft_consensus.cc:2258] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:38.942226 8623 tablet_replica.cc:330] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: stopping tablet replica
I20260929 16:01:38.943756 8624 ts_tablet_manager.cc:1905] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:38.944310 8623 raft_consensus.cc:2229] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:01:38.945082 8623 raft_consensus.cc:2258] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:01:38.949993 8626 ts_tablet_manager.cc:1918] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:38.950217 8626 log.cc:1196] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/f26db5ba08374b8bbf230664dc57f919
I20260929 16:01:38.950873 8626 ts_tablet_manager.cc:1939] T f26db5ba08374b8bbf230664dc57f919 P fc4712fa936540f5b0728b87379eca37: Deleting consensus metadata
I20260929 16:01:38.952345 8624 ts_tablet_manager.cc:1918] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:38.952879 8113 catalog_manager.cc:4841] TS fc4712fa936540f5b0728b87379eca37 (127.23.222.66:40511): tablet f26db5ba08374b8bbf230664dc57f919 (table test-table [id=440e17a75ef548e1a2b2cae18c83c445]) successfully deleted
I20260929 16:01:38.954334 8624 log.cc:1196] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/f26db5ba08374b8bbf230664dc57f919
I20260929 16:01:38.954511 8623 ts_tablet_manager.cc:1905] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:01:38.955034 8624 ts_tablet_manager.cc:1939] T f26db5ba08374b8bbf230664dc57f919 P fb59afc1f1bf44bea2e2cfc3b651b845: Deleting consensus metadata
I20260929 16:01:38.957238 8112 catalog_manager.cc:4841] TS fb59afc1f1bf44bea2e2cfc3b651b845 (127.23.222.65:42973): tablet f26db5ba08374b8bbf230664dc57f919 (table test-table [id=440e17a75ef548e1a2b2cae18c83c445]) successfully deleted
I20260929 16:01:38.960549 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:38.911908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39524 (local address 127.23.222.65:42973)
0929 16:01:38.912141 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:38.912150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:38.942619 (+ 30469us) server_negotiation.cc:407] Connection header received
0929 16:01:38.942842 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:38.942847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:38.942933 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:38.943057 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:38.945418 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.946655 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.948686 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:38.949366 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:38.949462 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:38.953160 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:38.953174 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:38.953192 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:38.953238 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:38.957228 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.958103 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.958114 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.958118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.958286 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:38.958773 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:38.958779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:38.958782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:38.959273 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:38.959604 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:38.959917 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 16:01:38.960152 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:01:38.961221 8309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:38.963302 8623 ts_tablet_manager.cc:1918] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:01:38.963617 8623 log.cc:1196] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/f26db5ba08374b8bbf230664dc57f919
I20260929 16:01:38.964614 8623 ts_tablet_manager.cc:1939] T f26db5ba08374b8bbf230664dc57f919 P 1a49823299094d45a46d74b35452764c: Deleting consensus metadata
I20260929 16:01:38.967041 8112 catalog_manager.cc:4841] TS 1a49823299094d45a46d74b35452764c (127.23.222.67:33435): tablet f26db5ba08374b8bbf230664dc57f919 (table test-table [id=440e17a75ef548e1a2b2cae18c83c445]) successfully deleted
I20260929 16:01:39.058617 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 8164
I20260929 16:01:39.084218 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 8318
I20260929 16:01:39.100354 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.084928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.23.222.66:40511)
0929 16:01:39.085099 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.085103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.085184 (+ 81us) server_negotiation.cc:407] Connection header received
0929 16:01:39.085383 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.085387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.085463 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.085553 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:39.087440 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.088134 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.090064 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.090550 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.090643 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.094113 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:39.094134 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:39.094149 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:39.094186 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:39.097282 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.098154 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.098161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.098164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.098327 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.098746 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.098752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.098755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.099105 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:39.099460 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.099810 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.100034 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:01:39.100992 8442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:39.191718 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 8318
I20260929 16:01:39.216737 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 8450
I20260929 16:01:39.235431 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:01:39.217518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.23.222.67:33435)
0929 16:01:39.217757 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 16:01:39.217765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:01:39.217838 (+ 73us) server_negotiation.cc:407] Connection header received
0929 16:01:39.218051 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:01:39.218057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:01:39.218132 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:01:39.218227 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:01:39.220372 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.221566 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.223528 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:01:39.224057 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:01:39.224174 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:01:39.227917 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:01:39.227933 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:01:39.227948 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:01:39.227994 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:01:39.231734 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.232884 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.232897 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.232900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.233109 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:01:39.233583 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:01:39.233586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:01:39.233588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:01:39.233971 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:01:39.234210 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 16:01:39.234673 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 16:01:39.234992 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:01:39.236254 8574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:39.362202 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 8450
I20260929 16:01:39.378929 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 8093
I20260929 16:01:39.380173 8157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:01:39.494036 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 8093
I20260929 16:01:59.516613 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37599
--webserver_interface=127.23.222.126
--webserver_port=42817
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:37599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:01:59.857637 8631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:01:59.858000 8631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:01:59.858160 8631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:01:59.865631 8631 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 16:01:59.865790 8631 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 16:01:59.865867 8631 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 16:01:59.865970 8631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:01:59.866074 8631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:01:59.866196 8631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:01:59.866287 8631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:01:59.866376 8631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:01:59.876884 8631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:37599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:37599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=42817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:01:59.878274 8631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:01:59.879838 8631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:01:59.892143 8636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:59.892123 8639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:01:59.892830 8637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:01:59.893443 8631 server_base.cc:1034] running on GCE node
I20260929 16:01:59.894269 8631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:01:59.895879 8631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:01:59.897125 8631 hybrid_clock.cc:648] HybridClock initialized: now 1790697719897052 us; error 60 us; skew 500 ppm
I20260929 16:01:59.897508 8631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:01:59 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697719, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:01:59.907390 8631 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:01:59.909118 8631 webserver.cc:460] Webserver started at http://127.23.222.126:42817/ using document root <none> and password file <none>
I20260929 16:01:59.909868 8631 fs_manager.cc:359] Metadata directory not provided
I20260929 16:01:59.910020 8631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:01:59.915822 8631 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:01:59.918658 8646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:01:59.920138 8631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:01:59.920351 8631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "73588ebc21114361a4f6fe399cc48cdd"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "7f296754d45deaceb3c5ca4a2f093d7e"
server_key_iv: "bc6afcbf64f3ae71852dfbedd5a9bcb2"
server_key_version: "encryptionkey@0"
I20260929 16:01:59.921291 8631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:01:59.944263 8631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:01:59.949012 8631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:01:59.949292 8631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:01:59.968973 8631 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:37599
I20260929 16:01:59.968993 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:37599 every 8 connection(s)
I20260929 16:01:59.970657 8631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:01:59.971777 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 8631
I20260929 16:01:59.972676 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:42973
--local_ip_for_outbound_sockets=127.23.222.65
--tserver_master_addrs=127.23.222.126:37599
--webserver_port=32789
--webserver_interface=127.23.222.65
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:01:59.978251 8699 sys_catalog.cc:263] Verifying existing consensus state
I20260929 16:01:59.982589 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Bootstrap starting.
I20260929 16:02:00.014611 8699 log.cc:824] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:00.036754 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 16:02:00.037384 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Bootstrap complete.
I20260929 16:02:00.046800 8699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:02:00.047837 8699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73588ebc21114361a4f6fe399cc48cdd, State: Initialized, Role: FOLLOWER
I20260929 16:02:00.048517 8699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:02:00.048789 8699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:00.048926 8699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:00.049118 8699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 2 FOLLOWER]: Advancing to term 3
I20260929 16:02:00.053035 8699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:02:00.053536 8699 leader_election.cc:302] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73588ebc21114361a4f6fe399cc48cdd; no voters:
I20260929 16:02:00.054282 8699 leader_election.cc:288] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 16:02:00.054430 8703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 16:02:00.055606 8703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [term 3 LEADER]: Becoming Leader. State: Replica: 73588ebc21114361a4f6fe399cc48cdd, State: Running, Role: LEADER
I20260929 16:02:00.056260 8703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } }
I20260929 16:02:00.056644 8699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:00.058678 8705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73588ebc21114361a4f6fe399cc48cdd. Latest consensus state: current_term: 3 leader_uuid: "73588ebc21114361a4f6fe399cc48cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } } }
I20260929 16:02:00.059075 8705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:00.059659 8704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "73588ebc21114361a4f6fe399cc48cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73588ebc21114361a4f6fe399cc48cdd" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 37599 } } }
I20260929 16:02:00.061092 8704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:00.068813 8709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:00.078779 8709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:00.080586 8709 catalog_manager.cc:1259] Loaded cluster ID: d6e40febac324fffa2e863424f1506b8
I20260929 16:02:00.080782 8709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:00.084800 8709 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:00.086408 8709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Loaded TSK: 0
I20260929 16:02:00.086582 8709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 73588ebc21114361a4f6fe399cc48cdd: Loaded TSK: 1
F20260929 16:02:00.087720 8709 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6344b829ad google::LogMessage::Fail() at ??:0
@ 0x7f6344b8588a google::LogMessage::SendToLog() at ??:0
@ 0x7f6344b824de google::LogMessage::Flush() at ??:0
@ 0x7f6344b83a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f63488e67d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f63488d573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f635a5b11c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f635a5b0783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f635a63c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f635a63c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f635a6865e9 std::function<>::operator()() at ??:0
@ 0x7f635a5b4cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f635a5b2c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f635a63a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f635a63a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f63477ce345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f63477d191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f63477d1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f63477aceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f634aa866db start_thread at ??:0
@ 0x7f6343a1c61f clone at ??:0
W20260929 16:02:00.389950 8701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:00.390298 8701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:00.390481 8701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:00.398116 8701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:00.398303 8701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:00.398502 8701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:02:00.409107 8701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:42973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=32789
--tserver_master_addrs=127.23.222.126:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:00.410604 8701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:00.412092 8701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:00.426548 8746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:00.426577 8747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:00.426663 8749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:00.427692 8701 server_base.cc:1034] running on GCE node
I20260929 16:02:00.428472 8701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:00.429621 8701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:00.430899 8701 hybrid_clock.cc:648] HybridClock initialized: now 1790697720430830 us; error 49 us; skew 500 ppm
I20260929 16:02:00.431329 8701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:00 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697720, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:00.434777 8701 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:02:00.436394 8701 webserver.cc:460] Webserver started at http://127.23.222.65:32789/ using document root <none> and password file <none>
I20260929 16:02:00.437088 8701 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:00.437193 8701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:00.442879 8701 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 16:02:00.445771 8756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:00.447486 8701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:00.447682 8701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "fb59afc1f1bf44bea2e2cfc3b651b845"
format_stamp: "Formatted at 2026-09-29 16:01:13 on dist-test-slave-cq88"
server_key: "22b536b1cc08478f648d03c391d33604"
server_key_iv: "5774bca8c059842e9eebe0b50dac0be0"
server_key_version: "encryptionkey@0"
I20260929 16:02:00.448719 8701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:00.464463 8701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:00.469450 8701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:00.469751 8701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:00.470875 8701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:00.472461 8701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:00.472561 8701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:00.472714 8701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:00.472813 8701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:00.508028 8701 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:42973
I20260929 16:02:00.508117 8869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:42973 every 8 connection(s)
I20260929 16:02:00.509564 8701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:00.514420 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:00.512590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:37191)
0929 16:02:00.513295 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:00.513536 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":267,"threads_started":1}
W20260929 16:02:00.515118 8870 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:37599 (0 consecutive failures): Network error: Failed to ping master at 127.23.222.126:37599: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
I20260929 16:02:00.515941 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:00.515629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:35613)
0929 16:02:00.515768 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:00.515837 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 16:02:00.516824 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:00.516445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:50963)
0929 16:02:00.516607 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:00.516700 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 16:02:00.517105 8870 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:02:00.517853 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 8701
I20260929 16:02:00.518527 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:40511
--local_ip_for_outbound_sockets=127.23.222.66
--tserver_master_addrs=127.23.222.126:37599
--webserver_port=42911
--webserver_interface=127.23.222.66
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:00.854781 8873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:00.855070 8873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:00.855165 8873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:00.862426 8873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:00.862535 8873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:00.862650 8873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:02:00.872535 8873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=42911
--tserver_master_addrs=127.23.222.126:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:00.873886 8873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:00.875262 8873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:00.888897 8878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:00.889106 8879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:00.889430 8881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:00.890214 8873 server_base.cc:1034] running on GCE node
I20260929 16:02:00.890830 8873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:00.891863 8873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:00.893128 8873 hybrid_clock.cc:648] HybridClock initialized: now 1790697720893036 us; error 41 us; skew 500 ppm
I20260929 16:02:00.893518 8873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:00 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697720, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:00.897004 8873 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:02:00.898663 8873 webserver.cc:460] Webserver started at http://127.23.222.66:42911/ using document root <none> and password file <none>
I20260929 16:02:00.899487 8873 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:00.899654 8873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:00.905519 8873 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:02:00.908433 8888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:00.909840 8873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:00.910035 8873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "fc4712fa936540f5b0728b87379eca37"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "42aeef3c5ce0e819beb3a2b84423a9c0"
server_key_iv: "1196a270605030c061dbd36450a35526"
server_key_version: "encryptionkey@0"
I20260929 16:02:00.911070 8873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:00.946221 8873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:00.950425 8873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:00.950716 8873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:00.951759 8873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:00.953305 8873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:00.953460 8873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:00.953593 8873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:00.953696 8873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:00.988503 8873 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:40511
I20260929 16:02:00.988632 9002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:40511 every 8 connection(s)
I20260929 16:02:00.990072 8873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:00.993523 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 8873
I20260929 16:02:00.994355 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:33435
--local_ip_for_outbound_sockets=127.23.222.67
--tserver_master_addrs=127.23.222.126:37599
--webserver_port=33269
--webserver_interface=127.23.222.67
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:02:00.995666 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:00.993470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:48683)
0929 16:02:00.994285 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:00.994634 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":333,"threads_started":1}
W20260929 16:02:00.996356 9003 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:37599 (0 consecutive failures): Network error: Failed to ping master at 127.23.222.126:37599: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
I20260929 16:02:00.997238 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:00.996907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:34125)
0929 16:02:00.997061 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:00.997137 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:02:00.998100 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:00.997763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:37833)
0929 16:02:00.997912 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:00.997992 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 16:02:00.998314 9003 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 16:02:01.338225 9006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:01.338534 9006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:01.338706 9006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:01.346495 9006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:01.346640 9006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:01.346827 9006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:02:01.357934 9006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:33435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=33269
--tserver_master_addrs=127.23.222.126:37599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:01.359429 9006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:01.360852 9006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:01.374608 9011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:01.374634 9012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:01.374755 9014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:01.375634 9006 server_base.cc:1034] running on GCE node
I20260929 16:02:01.376283 9006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:01.377326 9006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:01.378562 9006 hybrid_clock.cc:648] HybridClock initialized: now 1790697721378509 us; error 48 us; skew 500 ppm
I20260929 16:02:01.378921 9006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:01 dist-test-slave-cq88 krb5kdc[7530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697721, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:01.382296 9006 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:02:01.383891 9006 webserver.cc:460] Webserver started at http://127.23.222.67:33269/ using document root <none> and password file <none>
I20260929 16:02:01.384624 9006 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:01.384779 9006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:01.390220 9006 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 16:02:01.392988 9021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:01.394335 9006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:02:01.394508 9006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "1a49823299094d45a46d74b35452764c"
format_stamp: "Formatted at 2026-09-29 16:01:14 on dist-test-slave-cq88"
server_key: "9d2f009bb6f70fc941b52c8933ce3a18"
server_key_iv: "313197c773242466086ea7c0b5063580"
server_key_version: "encryptionkey@0"
I20260929 16:02:01.395584 9006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:01.410410 9006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:01.414582 9006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:01.414880 9006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:01.416019 9006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:01.417392 9006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:01.417515 9006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:01.417654 9006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:01.417770 9006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:01.452541 9006 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:33435
I20260929 16:02:01.452672 9133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:33435 every 8 connection(s)
I20260929 16:02:01.454042 9006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:01.458855 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:01.457060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:35265)
0929 16:02:01.457766 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:01.457993 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":281,"threads_started":1}
I20260929 16:02:01.459172 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 9006
W20260929 16:02:01.459721 9134 heartbeater.cc:644] Failed to heartbeat to 127.23.222.126:37599 (0 consecutive failures): Network error: Failed to ping master at 127.23.222.126:37599: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
I20260929 16:02:01.460736 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:01.460287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:47927)
0929 16:02:01.460499 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:01.460592 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 16:02:01.461709 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:01.461375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:42647)
0929 16:02:01.461519 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:01.461601 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:02:01.461930 9134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 16:02:01.518429 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:01.517611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:56123)
0929 16:02:01.518077 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:01.518181 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260929 16:02:01.999961 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:01.998924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:59121)
0929 16:02:01.999613 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:01.999720 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":431,"threads_started":1}
I20260929 16:02:02.463389 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.462482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:56773)
0929 16:02:02.462960 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:02.463072 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":240,"threads_started":1}
I20260929 16:02:02.520156 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:02.519376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:45045)
0929 16:02:02.519866 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:02.519964 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":236,"threads_started":1}
I20260929 16:02:03.001550 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:03.000837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:49323)
0929 16:02:03.001290 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:03.001375 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260929 16:02:03.465167 9143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:03.464318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:57129)
0929 16:02:03.464913 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:03.464997 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":339,"threads_started":1}
I20260929 16:02:03.521806 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:03.521083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:33995)
0929 16:02:03.521529 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:03.521623 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":248,"threads_started":1}
I20260929 16:02:04.003414 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:04.002628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:38697)
0929 16:02:04.003077 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:04.003163 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260929 16:02:04.467370 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:04.466469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:47847)
0929 16:02:04.466978 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:04.467072 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":264,"threads_started":1}
I20260929 16:02:04.523967 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:04.522982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:46917)
0929 16:02:04.523627 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:04.523751 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":329,"threads_started":1}
I20260929 16:02:05.005053 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:05.004352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:44065)
0929 16:02:05.004794 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:05.004879 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260929 16:02:05.469563 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:05.468483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:40923)
0929 16:02:05.469093 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:05.469262 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":332,"threads_started":1}
I20260929 16:02:05.526095 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:05.525194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:53461)
0929 16:02:05.525739 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:05.525846 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":305,"threads_started":1}
I20260929 16:02:06.006886 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:06.006104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:50389)
0929 16:02:06.006614 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:06.006708 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
I20260929 16:02:06.471474 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:06.470653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:42183)
0929 16:02:06.471136 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:06.471216 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":249,"threads_started":1}
I20260929 16:02:06.527918 9154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:06.527077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:55165)
0929 16:02:06.527634 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:06.527740 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":317,"threads_started":1}
I20260929 16:02:07.008831 9155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:07.007955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:58543)
0929 16:02:07.008524 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:07.008639 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":296,"threads_started":1}
I20260929 16:02:07.473243 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:07.472488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:58051)
0929 16:02:07.472975 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:07.473073 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":255,"threads_started":1}
I20260929 16:02:07.530041 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:07.529043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:51909)
0929 16:02:07.529716 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:07.529804 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":289,"threads_started":1}
I20260929 16:02:08.010768 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:08.009861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:53103)
0929 16:02:08.010443 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:08.010539 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":283,"threads_started":1}
I20260929 16:02:08.475402 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:08.474350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:34705)
0929 16:02:08.474945 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:08.475040 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":269,"threads_started":1}
I20260929 16:02:08.532183 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:08.531119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:48907)
0929 16:02:08.531906 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:08.532010 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":508,"threads_started":1}
I20260929 16:02:09.012708 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:09.011963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:34059)
0929 16:02:09.012455 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:09.012538 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":292,"threads_started":1}
I20260929 16:02:09.477313 9162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:09.476438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:36053)
0929 16:02:09.477002 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:09.477097 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":296,"threads_started":1}
I20260929 16:02:09.534066 9163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:09.533265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:52143)
0929 16:02:09.533786 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:09.533881 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":248,"threads_started":1}
I20260929 16:02:10.014675 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:10.013795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:41033)
0929 16:02:10.014359 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:10.014488 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":241,"threads_started":1}
I20260929 16:02:10.479025 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:10.478256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:44695)
0929 16:02:10.478758 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:10.478842 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":247,"threads_started":1}
I20260929 16:02:10.535915 9166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:10.535142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:57289)
0929 16:02:10.535641 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:10.535741 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":292,"threads_started":1}
I20260929 16:02:11.016669 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:11.015827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:39647)
0929 16:02:11.016404 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:11.016499 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":378,"threads_started":1}
I20260929 16:02:11.480918 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:11.480130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:54259)
0929 16:02:11.480631 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:11.480716 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":289,"threads_started":1}
I20260929 16:02:11.537904 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:11.537080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:40303)
0929 16:02:11.537609 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:11.537716 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":259,"threads_started":1}
I20260929 16:02:12.019141 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:12.017987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:32825)
0929 16:02:12.018787 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:12.018922 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":456,"threads_started":1}
I20260929 16:02:12.482803 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:12.481953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:46405)
0929 16:02:12.482470 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:12.482576 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":261,"threads_started":1}
I20260929 16:02:12.539816 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:12.538988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:35445)
0929 16:02:12.539536 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:12.539626 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":301,"threads_started":1}
I20260929 16:02:13.021240 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:13.020413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:42813)
0929 16:02:13.020937 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:13.021027 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":271,"threads_started":1}
I20260929 16:02:13.484694 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:13.483931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:55649)
0929 16:02:13.484437 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:13.484526 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":240,"threads_started":1}
I20260929 16:02:13.541872 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:13.541041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:48083)
0929 16:02:13.541556 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:13.541664 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":250,"threads_started":1}
I20260929 16:02:14.023036 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:14.022238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:34251)
0929 16:02:14.022738 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:14.022842 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":250,"threads_started":1}
I20260929 16:02:14.486671 9177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:14.485809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:33025)
0929 16:02:14.486377 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:14.486479 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":313,"threads_started":1}
I20260929 16:02:14.543650 9178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:14.542790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:60329)
0929 16:02:14.543363 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:14.543452 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":235,"threads_started":1}
I20260929 16:02:15.024801 9179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:15.024059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:43053)
0929 16:02:15.024526 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:15.024613 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260929 16:02:15.488494 9180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:15.487744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:43393)
0929 16:02:15.488200 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:15.488288 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":219,"threads_started":1}
I20260929 16:02:15.545763 9181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:15.544904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.65:38997)
0929 16:02:15.545478 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:15.545573 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":291,"threads_started":1}
I20260929 16:02:16.026893 9182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:16.026117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.66:55543)
0929 16:02:16.026575 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:16.026681 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
I20260929 16:02:16.469372 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 8701
I20260929 16:02:16.490593 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:16.489728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:37599 (local address 127.23.222.67:39817)
0929 16:02:16.490232 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:16.490332 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.126:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
I20260929 16:02:16.493904 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:16.470729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.23.222.65:42973)
0929 16:02:16.471933 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:16.471960 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:16.472015 (+ 55us) server_negotiation.cc:407] Connection header received
0929 16:02:16.473822 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:16.473833 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:16.474298 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:16.474620 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:16.476332 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:16.477660 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:16.479013 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:16.479789 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:16.479952 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:16.486831 (+ 6879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:16.486841 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:16.486861 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:16.486932 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:16.490406 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:16.491358 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:16.491366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:16.491374 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:16.491600 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:16.491970 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:16.491973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:16.491975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:16.492551 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:16.492785 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:16.493167 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 16:02:16.493610 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":977,"thread_start_us":325,"threads_started":1}
I20260929 16:02:16.495396 8865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:16.583509 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 8701
I20260929 16:02:16.601629 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 8873
I20260929 16:02:16.622602 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:16.602833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56856 (local address 127.23.222.66:40511)
0929 16:02:16.603474 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:16.603489 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:16.603551 (+ 62us) server_negotiation.cc:407] Connection header received
0929 16:02:16.605217 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:16.605228 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:16.605745 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:16.606078 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:16.607595 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:16.608897 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:16.610266 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:16.611519 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:16.611729 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:16.614818 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:16.614836 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:16.614856 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:16.614952 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:16.618460 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:16.619469 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:16.619477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:16.619485 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:16.619692 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:16.620190 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:16.620194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:16.620197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:16.620848 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:16.621222 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:16.621817 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 16:02:16.622303 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":286,"threads_started":1}
I20260929 16:02:16.624266 8997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:16.710410 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 8873
I20260929 16:02:16.728839 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 9006
I20260929 16:02:16.748549 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:16.729972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46902 (local address 127.23.222.67:33435)
0929 16:02:16.730615 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:16.730629 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:16.730671 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:02:16.732462 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:16.732472 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:16.732926 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:16.733303 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:16.734828 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:16.736207 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:16.737546 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:16.738241 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:16.738457 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:16.741451 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:16.741460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:16.741477 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:16.741522 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:16.744831 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:16.745685 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:16.745692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:16.745700 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:16.745889 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:16.746302 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:16.746307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:16.746311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:16.746910 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:16.747223 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:16.747755 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 16:02:16.748226 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":316,"threads_started":1}
I20260929 16:02:16.750023 9129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:16.835816 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 9006
2026-09-29T16:02:16Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65342 ms)
[----------] 30 tests from SecurityITest (305087 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:16 dist-test-slave-cq88 krb5kdc[9193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:16 dist-test-slave-cq88 krb5kdc[9193](info): set up 2 sockets
Sep 29 16:02:16 dist-test-slave-cq88 krb5kdc[9193](info): commencing operation
krb5kdc: starting...
W20260929 16:02:17.970883 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:18 dist-test-slave-cq88 krb5kdc[9193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:02:18.243219 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:46147
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:46147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:18.595147 9206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:18.595615 9206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:18.595772 9206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:18.603526 9206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:18.603688 9206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:18.603791 9206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:18.603892 9206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:18.603969 9206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:18.614420 9206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:46147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:46147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:18.615923 9206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:18.617475 9206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:18.630554 9214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:18.630663 9211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:18.630640 9212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:18.631599 9206 server_base.cc:1034] running on GCE node
I20260929 16:02:18.632462 9206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:18.633975 9206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:18.635217 9206 hybrid_clock.cc:648] HybridClock initialized: now 1790697738635174 us; error 52 us; skew 500 ppm
I20260929 16:02:18.635625 9206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:18 dist-test-slave-cq88 krb5kdc[9193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697738, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:18.638899 9206 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:02:18.640672 9206 webserver.cc:460] Webserver started at http://127.23.222.126:39103/ using document root <none> and password file <none>
I20260929 16:02:18.641399 9206 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:18.641508 9206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:18.641784 9206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:18.645115 9206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f"
format_stamp: "Formatted at 2026-09-29 16:02:18 on dist-test-slave-cq88"
server_key: "64082f02adc7ef9a475a981d181afb83"
server_key_iv: "c3be2088381451606ef4a3f229af41d2"
server_key_version: "encryptionkey@0"
I20260929 16:02:18.645938 9206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f"
format_stamp: "Formatted at 2026-09-29 16:02:18 on dist-test-slave-cq88"
server_key: "64082f02adc7ef9a475a981d181afb83"
server_key_iv: "c3be2088381451606ef4a3f229af41d2"
server_key_version: "encryptionkey@0"
I20260929 16:02:18.651819 9206 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 16:02:18.655696 9221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:18.657327 9206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:18.657558 9206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f"
format_stamp: "Formatted at 2026-09-29 16:02:18 on dist-test-slave-cq88"
server_key: "64082f02adc7ef9a475a981d181afb83"
server_key_iv: "c3be2088381451606ef4a3f229af41d2"
server_key_version: "encryptionkey@0"
I20260929 16:02:18.657781 9206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:18.673650 9206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:18.678836 9206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:18.679162 9206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:18.699864 9206 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:46147
I20260929 16:02:18.700034 9273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:46147 every 8 connection(s)
I20260929 16:02:18.701390 9206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:18.705799 9274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:18.709565 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 9206
I20260929 16:02:18.709810 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:18.710666 24441 external_mini_cluster.cc:1428] Setting key 4e22052887edc5b06d70b2373230d1a9
I20260929 16:02:18.720165 9274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f: Bootstrap starting.
I20260929 16:02:18.723841 9274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:18.724992 9274 log.cc:824] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:02:18 dist-test-slave-cq88 krb5kdc[9193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:18.728483 9274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f: No bootstrap required, opened a new log
I20260929 16:02:18.734951 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:18.713102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.23.222.126:46147)
0929 16:02:18.714034 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:18.714048 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:18.714095 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:02:18.715969 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:18.715996 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:18.716482 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:18.716945 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:18.718549 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:18.719995 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:18.721483 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:18.722210 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:18.722430 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:18.727091 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:18.727100 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:18.727122 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:18.727160 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:18.730400 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:18.731191 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:18.731204 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:18.731212 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:18.731461 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:18.731862 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:18.731867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:18.731874 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:18.732470 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:18.732815 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:18.733302 (+ 487us) server_negotiation.cc:299] Negotiation successful
0929 16:02:18.733799 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":402,"threads_started":1}
I20260929 16:02:18.736970 9274 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46147 } }
I20260929 16:02:18.737351 9274 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:18.737473 9274 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd5b126312ee4f9f87ae9d0b2ad0c53f, State: Initialized, Role: FOLLOWER
I20260929 16:02:18.738124 9274 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46147 } }
I20260929 16:02:18.738399 9274 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:18.738528 9274 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:18.738715 9274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:18.741082 9274 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46147 } }
I20260929 16:02:18.741604 9274 leader_election.cc:302] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd5b126312ee4f9f87ae9d0b2ad0c53f; no voters:
I20260929 16:02:18.742241 9274 leader_election.cc:288] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:18.742421 9279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:18.743136 9279 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [term 1 LEADER]: Becoming Leader. State: Replica: cd5b126312ee4f9f87ae9d0b2ad0c53f, State: Running, Role: LEADER
I20260929 16:02:18.743848 9279 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46147 } }
I20260929 16:02:18.744629 9274 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:18.746949 9280 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46147 } } }
I20260929 16:02:18.747391 9280 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:18.748261 9281 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd5b126312ee4f9f87ae9d0b2ad0c53f. Latest consensus state: current_term: 1 leader_uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd5b126312ee4f9f87ae9d0b2ad0c53f" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46147 } } }
I20260929 16:02:18.748641 9281 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:18.750634 9284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:18.756384 9284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:18.766541 9284 catalog_manager.cc:1347] Generated new cluster ID: 7cc69700b56847f795d9e82828c6985f
I20260929 16:02:18.766690 9284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:18.777359 9284 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:18.779398 9284 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:18.803445 9284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f: Generated new TSK 0
I20260929 16:02:18.804442 9284 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 16:02:18.876035 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46147
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:19.234127 9300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:19.234445 9300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:19.234549 9300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:19.242169 9300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:19.242295 9300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:19.242419 9300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:02:19.253163 9300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:19.254468 9300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:19.255824 9300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:19.270089 9305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:19.270025 9306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:19.270279 9308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:19.270762 9300 server_base.cc:1034] running on GCE node
I20260929 16:02:19.271525 9300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:19.272508 9300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:19.273732 9300 hybrid_clock.cc:648] HybridClock initialized: now 1790697739273683 us; error 57 us; skew 500 ppm
I20260929 16:02:19.274065 9300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:19 dist-test-slave-cq88 krb5kdc[9193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697739, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:19.277426 9300 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:02:19.278967 9300 webserver.cc:460] Webserver started at http://127.23.222.65:44379/ using document root <none> and password file <none>
I20260929 16:02:19.279773 9300 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:19.279919 9300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:19.280236 9300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:19.283355 9300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "20d093dca33e41d88b698ccdb7daa0ef"
format_stamp: "Formatted at 2026-09-29 16:02:19 on dist-test-slave-cq88"
server_key: "3293ed06cc8658197866c6b7dcc70320"
server_key_iv: "b3ec854b0812efb3d00a045588b8f16d"
server_key_version: "encryptionkey@0"
I20260929 16:02:19.284175 9300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "20d093dca33e41d88b698ccdb7daa0ef"
format_stamp: "Formatted at 2026-09-29 16:02:19 on dist-test-slave-cq88"
server_key: "3293ed06cc8658197866c6b7dcc70320"
server_key_iv: "b3ec854b0812efb3d00a045588b8f16d"
server_key_version: "encryptionkey@0"
I20260929 16:02:19.289832 9300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:02:19.293615 9315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:19.295050 9300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:02:19.295303 9300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "20d093dca33e41d88b698ccdb7daa0ef"
format_stamp: "Formatted at 2026-09-29 16:02:19 on dist-test-slave-cq88"
server_key: "3293ed06cc8658197866c6b7dcc70320"
server_key_iv: "b3ec854b0812efb3d00a045588b8f16d"
server_key_version: "encryptionkey@0"
I20260929 16:02:19.295552 9300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:19.308866 9300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:19.313746 9300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:19.314076 9300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:19.315148 9300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:19.316634 9300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:19.316745 9300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:19.316893 9300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:19.316965 9300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:19.352262 9300 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:44187
I20260929 16:02:19.352281 9428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:44187 every 8 connection(s)
I20260929 16:02:19.353659 9300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:19.362900 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 9300
I20260929 16:02:19.363196 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:19.363919 24441 external_mini_cluster.cc:1428] Setting key 18b9c72ce6ac7233524cec9df6ed290a
Sep 29 16:02:19 dist-test-slave-cq88 krb5kdc[9193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697739, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:19.379773 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:19.356573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:57567 (local address 127.23.222.126:46147)
0929 16:02:19.357064 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:19.357069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:19.358392 (+ 1323us) server_negotiation.cc:407] Connection header received
0929 16:02:19.360332 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:19.360337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:19.360430 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:19.360565 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:19.362247 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:19.363274 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:19.365172 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:19.365684 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:19.365844 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:19.370408 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:19.370418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:19.370425 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:19.370475 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:19.375144 (+ 4669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:19.376562 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:19.376569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:19.376590 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:19.376774 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:19.377406 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:19.377414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:19.377419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:19.377844 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:19.378118 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:19.379046 (+ 928us) server_negotiation.cc:299] Negotiation successful
0929 16:02:19.379493 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260929 16:02:19.381196 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:19.357172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:46147 (local address 127.23.222.65:57567)
0929 16:02:19.358176 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:19.358256 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:19.360030 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:19.360839 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:19.360847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:19.361443 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:19.362054 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:19.362073 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:19.363435 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:19.363441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:19.365020 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:19.365033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:19.365963 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:19.365971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:19.366281 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:19.367087 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:19.367124 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:19.370102 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:19.375417 (+ 5315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:19.375430 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:19.375456 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:19.376297 (+ 841us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:19.376950 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:19.376958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:19.376964 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:19.377235 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:19.378009 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:19.378021 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:19.378528 (+ 507us) client_negotiation.cc:769] Sending connection context
0929 16:02:19.378867 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 16:02:19.379565 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":359,"threads_started":1}
I20260929 16:02:19.383001 9429 heartbeater.cc:347] Connected to a master server at 127.23.222.126:46147
I20260929 16:02:19.383399 9429 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:19.384415 9429 heartbeater.cc:511] Master 127.23.222.126:46147 requested a full tablet report, sending...
I20260929 16:02:19.386909 9239 ts_manager.cc:194] Registered new tserver with Master: 20d093dca33e41d88b698ccdb7daa0ef (127.23.222.65:44187)
I20260929 16:02:19.388528 9239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:57567
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 16:02:19.427630 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46147
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:19.780040 9435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:19.780351 9435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:19.780519 9435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:19.788123 9435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:19.788249 9435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:19.788471 9435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:02:19.798753 9435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:19.800313 9435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:19.801783 9435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:19.815838 9440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:19.815774 9441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:19.815815 9443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:19.816649 9435 server_base.cc:1034] running on GCE node
I20260929 16:02:19.817282 9435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:19.818248 9435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:19.819458 9435 hybrid_clock.cc:648] HybridClock initialized: now 1790697739819399 us; error 54 us; skew 500 ppm
I20260929 16:02:19.819844 9435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:19 dist-test-slave-cq88 krb5kdc[9193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697739, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:19.823072 9435 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:02:19.824616 9435 webserver.cc:460] Webserver started at http://127.23.222.66:33775/ using document root <none> and password file <none>
I20260929 16:02:19.825318 9435 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:19.825430 9435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:19.825704 9435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:19.828857 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "6cc8d419f72d4b258e5b430627d6eeb7"
format_stamp: "Formatted at 2026-09-29 16:02:19 on dist-test-slave-cq88"
server_key: "d7f66d89850e5ca2a5359d14841cacc0"
server_key_iv: "13a555e7e74c89d4f5def18a430a2dc5"
server_key_version: "encryptionkey@0"
I20260929 16:02:19.829622 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "6cc8d419f72d4b258e5b430627d6eeb7"
format_stamp: "Formatted at 2026-09-29 16:02:19 on dist-test-slave-cq88"
server_key: "d7f66d89850e5ca2a5359d14841cacc0"
server_key_iv: "13a555e7e74c89d4f5def18a430a2dc5"
server_key_version: "encryptionkey@0"
I20260929 16:02:19.835150 9435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:02:19.838930 9451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:19.840418 9435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:19.840624 9435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "6cc8d419f72d4b258e5b430627d6eeb7"
format_stamp: "Formatted at 2026-09-29 16:02:19 on dist-test-slave-cq88"
server_key: "d7f66d89850e5ca2a5359d14841cacc0"
server_key_iv: "13a555e7e74c89d4f5def18a430a2dc5"
server_key_version: "encryptionkey@0"
I20260929 16:02:19.840832 9435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:19.858217 9435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:19.862955 9435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:19.863283 9435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:19.864241 9435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:19.865900 9435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:19.865984 9435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:19.866079 9435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:19.866124 9435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:19.904541 9435 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:32821
I20260929 16:02:19.904583 9564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:32821 every 8 connection(s)
I20260929 16:02:19.905921 9435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:19.913692 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 9435
I20260929 16:02:19.913897 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:19.914366 24441 external_mini_cluster.cc:1428] Setting key fddc47a3af2476888f1fb73eae3686ea
Sep 29 16:02:19 dist-test-slave-cq88 krb5kdc[9193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697739, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:19.928373 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:19.908631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:60833 (local address 127.23.222.126:46147)
0929 16:02:19.909107 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:19.909114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:19.910228 (+ 1114us) server_negotiation.cc:407] Connection header received
0929 16:02:19.912178 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:19.912183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:19.912260 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:19.912370 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:19.913967 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:19.914778 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:19.916649 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:19.917085 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:19.917221 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:19.921497 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:19.921503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:19.921507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:19.921545 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:19.924565 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:19.925485 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:19.925492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:19.925497 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:19.925678 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:19.926397 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:19.926401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:19.926403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:19.926723 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:19.926863 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:19.927898 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0929 16:02:19.928095 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260929 16:02:19.930012 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:19.909333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:46147 (local address 127.23.222.66:60833)
0929 16:02:19.910063 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:19.910100 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:19.911888 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:19.912600 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:19.912610 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:19.913158 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:19.913743 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:19.913788 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:19.914952 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:19.914958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:19.916502 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:19.916514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:19.917326 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:19.917334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:19.917583 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:19.918314 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:19.918337 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:19.921210 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:19.924768 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:19.924777 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:19.924794 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:19.925323 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:19.925838 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:19.925845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:19.925852 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:19.926164 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:19.926875 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:19.926888 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:19.927453 (+ 565us) client_negotiation.cc:769] Sending connection context
0929 16:02:19.927781 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 16:02:19.928435 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":288,"threads_started":1}
I20260929 16:02:19.931615 9565 heartbeater.cc:347] Connected to a master server at 127.23.222.126:46147
I20260929 16:02:19.931945 9565 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:19.932744 9565 heartbeater.cc:511] Master 127.23.222.126:46147 requested a full tablet report, sending...
I20260929 16:02:19.934556 9239 ts_manager.cc:194] Registered new tserver with Master: 6cc8d419f72d4b258e5b430627d6eeb7 (127.23.222.66:32821)
I20260929 16:02:19.935967 9239 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:60833
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 16:02:19.976094 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46147
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:37469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:20.326819 9571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:20.327210 9571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:20.327412 9571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:20.335013 9571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:20.335196 9571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:20.335407 9571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:02:20.346318 9571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:37469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:20.347839 9571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:20.349238 9571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:20.366916 9576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:20.366809 9577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:20.366858 9579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:20.368167 9571 server_base.cc:1034] running on GCE node
I20260929 16:02:20.368866 9571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:20.369861 9571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:20.371115 9571 hybrid_clock.cc:648] HybridClock initialized: now 1790697740371040 us; error 63 us; skew 500 ppm
I20260929 16:02:20.371634 9571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:20 dist-test-slave-cq88 krb5kdc[9193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697740, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:20.375531 9571 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:02:20.377169 9571 webserver.cc:460] Webserver started at http://127.23.222.67:44787/ using document root <none> and password file <none>
I20260929 16:02:20.377952 9571 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:20.378108 9571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:20.378443 9571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:20.381726 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "8c9018af9bec4d6683a82af35b46236c"
format_stamp: "Formatted at 2026-09-29 16:02:20 on dist-test-slave-cq88"
server_key: "c3a317a5179b7ba61a979ca1833d0f7e"
server_key_iv: "dd4ef6200c730250a6bb4709163dca6b"
server_key_version: "encryptionkey@0"
I20260929 16:02:20.382573 9571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "8c9018af9bec4d6683a82af35b46236c"
format_stamp: "Formatted at 2026-09-29 16:02:20 on dist-test-slave-cq88"
server_key: "c3a317a5179b7ba61a979ca1833d0f7e"
server_key_iv: "dd4ef6200c730250a6bb4709163dca6b"
server_key_version: "encryptionkey@0"
I20260929 16:02:20.388435 9571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 16:02:20.392815 9429 heartbeater.cc:503] Master 127.23.222.126:46147 was elected leader, sending a full tablet report...
I20260929 16:02:20.393077 9586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:20.395499 9571 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260929 16:02:20.395699 9571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "8c9018af9bec4d6683a82af35b46236c"
format_stamp: "Formatted at 2026-09-29 16:02:20 on dist-test-slave-cq88"
server_key: "c3a317a5179b7ba61a979ca1833d0f7e"
server_key_iv: "dd4ef6200c730250a6bb4709163dca6b"
server_key_version: "encryptionkey@0"
I20260929 16:02:20.395913 9571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:20.418712 9571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:20.423684 9571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:20.424002 9571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:20.425081 9571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:20.426576 9571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:20.426664 9571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:20.426798 9571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:20.426853 9571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:20.462036 9571 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:32945
I20260929 16:02:20.462124 9699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:32945 every 8 connection(s)
I20260929 16:02:20.463670 9571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:20.472442 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 9571
I20260929 16:02:20.472646 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:20.473202 24441 external_mini_cluster.cc:1428] Setting key e9893d8f3db1518c30bdb68ba9172554
Sep 29 16:02:20 dist-test-slave-cq88 krb5kdc[9193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697740, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:20.486554 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.466499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:38463 (local address 127.23.222.126:46147)
0929 16:02:20.467050 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:20.467056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:20.468021 (+ 965us) server_negotiation.cc:407] Connection header received
0929 16:02:20.469999 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:20.470004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:20.470099 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:20.470208 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:20.472060 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.473339 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.475685 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.476127 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.476241 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:20.480146 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:20.480153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:20.480157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:20.480210 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:20.483126 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:20.484150 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:20.484155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:20.484158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:20.484304 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:20.484784 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:20.484788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:20.484790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:20.485104 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:20.485225 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:20.486052 (+ 827us) server_negotiation.cc:299] Negotiation successful
0929 16:02:20.486260 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":261,"threads_started":1}
I20260929 16:02:20.487690 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.467016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:46147 (local address 127.23.222.67:38463)
0929 16:02:20.467838 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:20.467880 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:20.469658 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:20.470473 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:20.470483 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:20.471067 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:20.471793 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:20.471818 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.473561 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.473566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:20.475526 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:20.475542 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.476386 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.476393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:20.476728 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:20.477382 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:20.477406 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:20.479838 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:20.483381 (+ 3543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:20.483390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:20.483406 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:20.483975 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:20.484420 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:20.484424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:20.484427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:20.484634 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:20.485243 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:20.485250 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:20.485637 (+ 387us) client_negotiation.cc:769] Sending connection context
0929 16:02:20.485897 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 16:02:20.486340 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":323,"threads_started":1}
I20260929 16:02:20.489724 9700 heartbeater.cc:347] Connected to a master server at 127.23.222.126:46147
I20260929 16:02:20.490155 9700 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:20.490945 9700 heartbeater.cc:511] Master 127.23.222.126:46147 requested a full tablet report, sending...
I20260929 16:02:20.492719 9238 ts_manager.cc:194] Registered new tserver with Master: 8c9018af9bec4d6683a82af35b46236c (127.23.222.67:32945)
I20260929 16:02:20.493762 9238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:38463
I20260929 16:02:20.502312 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:20.521198 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.507217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.23.222.126:46147)
0929 16:02:20.507445 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:20.507450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:20.507951 (+ 501us) server_negotiation.cc:407] Connection header received
0929 16:02:20.508111 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:20.508115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:20.508186 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:20.508277 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:20.509795 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.510587 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.512266 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.512765 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.512884 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:20.515591 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:20.515597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:20.515603 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:20.515649 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:20.518357 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:20.519164 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:20.519169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:20.519171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:20.519402 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:20.519813 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:20.519817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:20.519819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:20.520188 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:20.520445 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:20.520745 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 16:02:20.520946 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 16:02:20.526489 9238 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:20.534138 9238 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:20.559057 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.547733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:32821 (local address 127.0.0.1:37090)
0929 16:02:20.549260 (+ 1527us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:20.549294 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:20.549452 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:20.550450 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:20.550460 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:20.550496 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:20.550931 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:20.550945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.553231 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.553240 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:20.555790 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:20.555808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.557832 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.557840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:20.558106 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:20.558167 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 16:02:20.558353 (+ 186us) client_negotiation.cc:240] Negotiation successful
0929 16:02:20.558593 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"mutex_wait_us":23,"spinlock_wait_cycles":5504,"thread_start_us":1209,"threads_started":1}
I20260929 16:02:20.560141 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.548137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.23.222.66:32821)
0929 16:02:20.548991 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:20.549001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:20.549414 (+ 413us) server_negotiation.cc:407] Connection header received
0929 16:02:20.549657 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:20.549666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:20.549933 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:20.550157 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:20.551135 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.553030 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.555957 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.557577 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.557729 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:20.557961 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:20.559604 (+ 1643us) server_negotiation.cc:299] Negotiation successful
0929 16:02:20.559800 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":482,"threads_started":1}
I20260929 16:02:20.561640 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.550332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:32945 (local address 127.0.0.1:52858)
0929 16:02:20.553574 (+ 3242us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:20.553595 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:20.553731 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:20.554355 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:20.554359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:20.554387 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:20.554860 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:20.554873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.556648 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.556653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:20.558394 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:20.558410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.560745 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.560749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:20.560882 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:20.560903 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 16:02:20.561040 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 16:02:20.561201 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3063,"mutex_wait_us":1,"thread_start_us":2861,"threads_started":1}
I20260929 16:02:20.562071 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.549755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:44187 (local address 127.0.0.1:46108)
0929 16:02:20.550363 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:20.550387 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:20.550544 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:20.553322 (+ 2778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:20.553327 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:20.553361 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:20.553915 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:20.553928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.555990 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.555995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:20.558188 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:20.558200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.561159 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.561165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:20.561353 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:20.561388 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:02:20.561494 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 16:02:20.561660 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":28,"spinlock_wait_cycles":4992,"thread_start_us":285,"threads_started":1}
I20260929 16:02:20.562110 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.549994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46108 (local address 127.23.222.65:44187)
0929 16:02:20.552651 (+ 2657us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:20.552662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:20.552692 (+ 30us) server_negotiation.cc:407] Connection header received
0929 16:02:20.552803 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:20.552811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:20.553059 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:20.553966 (+ 907us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:20.554198 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.555879 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.558359 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.560957 (+ 2598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.561095 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:20.561324 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:20.561560 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 16:02:20.561740 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2472,"thread_start_us":2338,"threads_started":1}
I20260929 16:02:20.562819 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.550915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.23.222.67:32945)
0929 16:02:20.553018 (+ 2103us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:20.553027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:20.553713 (+ 686us) server_negotiation.cc:407] Connection header received
0929 16:02:20.553909 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:20.553916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:20.554146 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:20.554335 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:20.555084 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.556504 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.558582 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:20.560221 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:20.560338 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:20.560575 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:20.562258 (+ 1683us) server_negotiation.cc:299] Negotiation successful
0929 16:02:20.562456 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":1801,"threads_started":1}
I20260929 16:02:20.564503 9499 tablet_service.cc:1467] Processing CreateTablet for tablet 2d93019c7fe24b00a3ca1d237721b21a (DEFAULT_TABLE table=test-table [id=42bbd541d281479497d5533ff4546c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:20.565776 9364 tablet_service.cc:1467] Processing CreateTablet for tablet 2d93019c7fe24b00a3ca1d237721b21a (DEFAULT_TABLE table=test-table [id=42bbd541d281479497d5533ff4546c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:20.566996 9364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d93019c7fe24b00a3ca1d237721b21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:20.567041 9499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d93019c7fe24b00a3ca1d237721b21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:20.568930 9635 tablet_service.cc:1467] Processing CreateTablet for tablet 2d93019c7fe24b00a3ca1d237721b21a (DEFAULT_TABLE table=test-table [id=42bbd541d281479497d5533ff4546c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:20.570552 9635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d93019c7fe24b00a3ca1d237721b21a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:20.580749 9719 tablet_bootstrap.cc:492] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: Bootstrap starting.
I20260929 16:02:20.584079 9720 tablet_bootstrap.cc:492] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: Bootstrap starting.
I20260929 16:02:20.584802 9719 tablet_bootstrap.cc:654] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:20.586349 9719 log.cc:824] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:20.588393 9720 tablet_bootstrap.cc:654] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:20.589905 9722 tablet_bootstrap.cc:492] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: Bootstrap starting.
I20260929 16:02:20.589960 9720 log.cc:824] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:20.590543 9719 tablet_bootstrap.cc:492] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: No bootstrap required, opened a new log
I20260929 16:02:20.590958 9719 ts_tablet_manager.cc:1397] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260929 16:02:20.593750 9722 tablet_bootstrap.cc:654] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:20.594215 9720 tablet_bootstrap.cc:492] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: No bootstrap required, opened a new log
I20260929 16:02:20.594517 9720 ts_tablet_manager.cc:1397] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260929 16:02:20.595129 9722 log.cc:824] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:20.598742 9722 tablet_bootstrap.cc:492] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: No bootstrap required, opened a new log
I20260929 16:02:20.599172 9722 ts_tablet_manager.cc:1397] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 16:02:20.600112 9719 raft_consensus.cc:348] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } }
I20260929 16:02:20.600648 9719 raft_consensus.cc:374] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:20.600764 9719 raft_consensus.cc:729] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d093dca33e41d88b698ccdb7daa0ef, State: Initialized, Role: FOLLOWER
I20260929 16:02:20.601446 9719 consensus_queue.cc:260] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } }
I20260929 16:02:20.603178 9719 ts_tablet_manager.cc:1428] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260929 16:02:20.606253 9430 tablet.cc:2369] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:20.606081 9720 raft_consensus.cc:348] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } }
I20260929 16:02:20.606596 9720 raft_consensus.cc:374] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:20.606709 9720 raft_consensus.cc:729] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc8d419f72d4b258e5b430627d6eeb7, State: Initialized, Role: FOLLOWER
I20260929 16:02:20.607422 9720 consensus_queue.cc:260] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } }
I20260929 16:02:20.608872 9565 heartbeater.cc:503] Master 127.23.222.126:46147 was elected leader, sending a full tablet report...
I20260929 16:02:20.609444 9720 ts_tablet_manager.cc:1428] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: Time spent starting tablet: real 0.015s user 0.012s sys 0.003s
I20260929 16:02:20.609913 9722 raft_consensus.cc:348] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } }
I20260929 16:02:20.610355 9722 raft_consensus.cc:374] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:20.610464 9722 raft_consensus.cc:729] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9018af9bec4d6683a82af35b46236c, State: Initialized, Role: FOLLOWER
I20260929 16:02:20.611088 9722 consensus_queue.cc:260] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } }
I20260929 16:02:20.612445 9700 heartbeater.cc:503] Master 127.23.222.126:46147 was elected leader, sending a full tablet report...
I20260929 16:02:20.613108 9722 ts_tablet_manager.cc:1428] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
W20260929 16:02:20.658145 9566 tablet.cc:2369] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:20.715459 9701 tablet.cc:2369] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:20.995569 9725 raft_consensus.cc:482] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:20.995913 9725 raft_consensus.cc:504] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } }
I20260929 16:02:20.997634 9725 leader_election.cc:288] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc8d419f72d4b258e5b430627d6eeb7 (127.23.222.66:32821), 8c9018af9bec4d6683a82af35b46236c (127.23.222.67:32945)
I20260929 16:02:21.004505 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.998161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:54139 (local address 127.23.222.66:32821)
0929 16:02:20.998386 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:20.998392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:20.998655 (+ 263us) server_negotiation.cc:407] Connection header received
0929 16:02:20.998936 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:20.998942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:20.999019 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:20.999110 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:20.999820 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.000650 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.002451 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.003417 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.003618 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:21.003723 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:21.004064 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:02:21.004221 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:02:21.004608 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.997925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:32821 (local address 127.23.222.65:54139)
0929 16:02:20.998522 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:20.998539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:20.998783 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:20.999188 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:20.999193 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:20.999219 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:20.999605 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:20.999614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.000781 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.000789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:21.002285 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:21.002301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.003637 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.003643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:21.003835 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:21.003861 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:02:21.004023 (+ 162us) client_negotiation.cc:240] Negotiation successful
0929 16:02:21.004162 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":284,"threads_started":1}
I20260929 16:02:21.005568 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.998851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:34309 (local address 127.23.222.67:32945)
0929 16:02:20.999073 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:20.999081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:20.999657 (+ 576us) server_negotiation.cc:407] Connection header received
0929 16:02:20.999894 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:20.999899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:20.999970 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:21.000066 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:21.000638 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.001481 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.003198 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.004381 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.004558 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:21.004636 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:21.005269 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 16:02:21.005387 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:02:21.005774 9519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93019c7fe24b00a3ca1d237721b21a" candidate_uuid: "20d093dca33e41d88b698ccdb7daa0ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" is_pre_election: true
I20260929 16:02:21.006111 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:20.998692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:32945 (local address 127.23.222.65:34309)
0929 16:02:20.999515 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:20.999569 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:20.999753 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:21.000092 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:21.000096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:21.000121 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:21.000491 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:21.000507 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.001618 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.001624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:21.003021 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:21.003032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.004823 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.004832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:21.005027 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:21.005062 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 16:02:21.005216 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 16:02:21.005861 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":417,"threads_started":1}
I20260929 16:02:21.006417 9519 raft_consensus.cc:2454] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d093dca33e41d88b698ccdb7daa0ef in term 0.
I20260929 16:02:21.007066 9655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93019c7fe24b00a3ca1d237721b21a" candidate_uuid: "20d093dca33e41d88b698ccdb7daa0ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9018af9bec4d6683a82af35b46236c" is_pre_election: true
I20260929 16:02:21.007138 9318 leader_election.cc:302] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d093dca33e41d88b698ccdb7daa0ef, 6cc8d419f72d4b258e5b430627d6eeb7; no voters:
I20260929 16:02:21.007557 9655 raft_consensus.cc:2454] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20d093dca33e41d88b698ccdb7daa0ef in term 0.
I20260929 16:02:21.007601 9725 raft_consensus.cc:2781] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:21.007745 9725 raft_consensus.cc:482] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:21.007862 9725 raft_consensus.cc:3037] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:21.010119 9725 raft_consensus.cc:504] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } }
I20260929 16:02:21.010864 9725 leader_election.cc:288] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [CANDIDATE]: Term 1 election: Requested vote from peers 6cc8d419f72d4b258e5b430627d6eeb7 (127.23.222.66:32821), 8c9018af9bec4d6683a82af35b46236c (127.23.222.67:32945)
I20260929 16:02:21.011488 9519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93019c7fe24b00a3ca1d237721b21a" candidate_uuid: "20d093dca33e41d88b698ccdb7daa0ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc8d419f72d4b258e5b430627d6eeb7"
I20260929 16:02:21.011576 9655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d93019c7fe24b00a3ca1d237721b21a" candidate_uuid: "20d093dca33e41d88b698ccdb7daa0ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c9018af9bec4d6683a82af35b46236c"
I20260929 16:02:21.011744 9519 raft_consensus.cc:3037] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:21.011812 9655 raft_consensus.cc:3037] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:21.015208 9519 raft_consensus.cc:2454] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d093dca33e41d88b698ccdb7daa0ef in term 1.
I20260929 16:02:21.015208 9655 raft_consensus.cc:2454] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20d093dca33e41d88b698ccdb7daa0ef in term 1.
I20260929 16:02:21.015758 9318 leader_election.cc:302] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d093dca33e41d88b698ccdb7daa0ef, 8c9018af9bec4d6683a82af35b46236c; no voters:
I20260929 16:02:21.016084 9725 raft_consensus.cc:2781] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:21.016592 9725 raft_consensus.cc:686] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 1 LEADER]: Becoming Leader. State: Replica: 20d093dca33e41d88b698ccdb7daa0ef, State: Running, Role: LEADER
I20260929 16:02:21.017118 9725 consensus_queue.cc:237] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } }
I20260929 16:02:21.022318 9239 catalog_manager.cc:5495] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 20d093dca33e41d88b698ccdb7daa0ef (127.23.222.65). New cstate: current_term: 1 leader_uuid: "20d093dca33e41d88b698ccdb7daa0ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20d093dca33e41d88b698ccdb7daa0ef" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 44187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 } health_report { overall_health: UNKNOWN } } }
I20260929 16:02:21.128331 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:21.122303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.23.222.65:44187)
0929 16:02:21.122969 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:21.122977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:21.123002 (+ 25us) server_negotiation.cc:407] Connection header received
0929 16:02:21.123113 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:21.123116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:21.123196 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:21.123426 (+ 230us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:21.123827 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.124547 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.126020 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.126446 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.126587 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:21.127095 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:21.127703 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:21.127876 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:21.127930 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 16:02:21.128017 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":460,"threads_started":1}
I20260929 16:02:21.137657 9655 raft_consensus.cc:1261] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [term 1 FOLLOWER]: Refusing update from remote peer 20d093dca33e41d88b698ccdb7daa0ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:21.137656 9519 raft_consensus.cc:1261] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [term 1 FOLLOWER]: Refusing update from remote peer 20d093dca33e41d88b698ccdb7daa0ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:21.138796 9725 consensus_queue.cc:1035] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9018af9bec4d6683a82af35b46236c" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:21.139267 9730 consensus_queue.cc:1035] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc8d419f72d4b258e5b430627d6eeb7" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 32821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:21.157447 9735 mvcc.cc:205] Tried to move back new op lower bound from 7334697947686129664 to 7334697947209932800. Current Snapshot: MvccSnapshot[applied={T|T < 7334697947686129664}]
I20260929 16:02:21.161619 9738 mvcc.cc:205] Tried to move back new op lower bound from 7334697947686129664 to 7334697947209932800. Current Snapshot: MvccSnapshot[applied={T|T < 7334697947686129664}]
I20260929 16:02:21.176239 9238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:21.176569 9238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:21.181097 9238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd5b126312ee4f9f87ae9d0b2ad0c53f: Sending DeleteTablet for 3 replicas of tablet 2d93019c7fe24b00a3ca1d237721b21a
I20260929 16:02:21.182516 9499 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93019c7fe24b00a3ca1d237721b21a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:21 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:37090
I20260929 16:02:21.182845 9635 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93019c7fe24b00a3ca1d237721b21a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:21 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:52858
I20260929 16:02:21.182922 9364 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93019c7fe24b00a3ca1d237721b21a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:21 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:46108
I20260929 16:02:21.183333 9634 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d93019c7fe24b00a3ca1d237721b21a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:21 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:52858
I20260929 16:02:21.184892 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 9300
I20260929 16:02:21.197965 9749 tablet_replica.cc:330] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: stopping tablet replica
I20260929 16:02:21.198499 9749 raft_consensus.cc:2229] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:21.199075 9749 raft_consensus.cc:2258] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:21.203310 9749 ts_tablet_manager.cc:1905] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:21.210994 9749 ts_tablet_manager.cc:1918] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:21.211218 9749 log.cc:1196] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/2d93019c7fe24b00a3ca1d237721b21a
I20260929 16:02:21.214623 9748 tablet_replica.cc:330] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: stopping tablet replica
I20260929 16:02:21.214888 9749 ts_tablet_manager.cc:1939] T 2d93019c7fe24b00a3ca1d237721b21a P 8c9018af9bec4d6683a82af35b46236c: Deleting consensus metadata
I20260929 16:02:21.215364 9748 raft_consensus.cc:2229] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:21.216048 9750 tablet_replica.cc:330] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: stopping tablet replica
I20260929 16:02:21.216482 9750 raft_consensus.cc:2229] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:21.217092 9224 catalog_manager.cc:4841] TS 8c9018af9bec4d6683a82af35b46236c (127.23.222.67:32945): tablet 2d93019c7fe24b00a3ca1d237721b21a (table test-table [id=42bbd541d281479497d5533ff4546c8e]) successfully deleted
I20260929 16:02:21.217166 9750 raft_consensus.cc:2258] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 16:02:21.217659 9224 catalog_manager.cc:4808] TS 8c9018af9bec4d6683a82af35b46236c (127.23.222.67:32945): delete failed for tablet 2d93019c7fe24b00a3ca1d237721b21a because the tablet was not found. No further retry: Not found: Tablet not found: 2d93019c7fe24b00a3ca1d237721b21a
I20260929 16:02:21.217962 9748 raft_consensus.cc:2258] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:21.219290 9750 ts_tablet_manager.cc:1905] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:21.220422 9748 ts_tablet_manager.cc:1905] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:21.226694 9750 ts_tablet_manager.cc:1918] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:21.226899 9750 log.cc:1196] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/2d93019c7fe24b00a3ca1d237721b21a
I20260929 16:02:21.227516 9750 ts_tablet_manager.cc:1939] T 2d93019c7fe24b00a3ca1d237721b21a P 20d093dca33e41d88b698ccdb7daa0ef: Deleting consensus metadata
I20260929 16:02:21.229709 9224 catalog_manager.cc:4841] TS 20d093dca33e41d88b698ccdb7daa0ef (127.23.222.65:44187): tablet 2d93019c7fe24b00a3ca1d237721b21a (table test-table [id=42bbd541d281479497d5533ff4546c8e]) successfully deleted
Sep 29 16:02:21 dist-test-slave-cq88 krb5kdc[9193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 16:02:21.235730 9748 ts_tablet_manager.cc:1918] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:21.235965 9748 log.cc:1196] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/2d93019c7fe24b00a3ca1d237721b21a
I20260929 16:02:21.236506 9748 ts_tablet_manager.cc:1939] T 2d93019c7fe24b00a3ca1d237721b21a P 6cc8d419f72d4b258e5b430627d6eeb7: Deleting consensus metadata
I20260929 16:02:21.238631 9224 catalog_manager.cc:4841] TS 6cc8d419f72d4b258e5b430627d6eeb7 (127.23.222.66:32821): tablet 2d93019c7fe24b00a3ca1d237721b21a (table test-table [id=42bbd541d281479497d5533ff4546c8e]) successfully deleted
I20260929 16:02:21.239149 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:21.186697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.23.222.65:44187)
0929 16:02:21.186943 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:21.186953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:21.199646 (+ 12693us) server_negotiation.cc:407] Connection header received
0929 16:02:21.199998 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:21.200007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:21.200148 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:21.200282 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:21.202485 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.204130 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.213917 (+ 9787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.214951 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.215156 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:21.231908 (+ 16752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:21.231923 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:21.231938 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:21.231986 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:21.235413 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:21.236474 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:21.236487 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:21.236492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:21.236717 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:21.237206 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:21.237211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:21.237214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:21.237677 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:21.238012 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:21.238395 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 16:02:21.238714 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:02:21.240060 9424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:21.340248 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 9300
I20260929 16:02:21.361071 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 9435
Sep 29 16:02:21 dist-test-slave-cq88 krb5kdc[9193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 16:02:21.377664 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:21.361943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.23.222.66:32821)
0929 16:02:21.362133 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:21.362138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:21.362157 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:02:21.362351 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:21.362355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:21.362433 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:21.362522 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:21.364038 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.364782 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.366178 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.366604 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.366702 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:21.371690 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:21.371707 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:21.371722 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:21.371774 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:21.374768 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:21.375620 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:21.375628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:21.375630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:21.375782 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:21.376134 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:21.376138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:21.376140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:21.376578 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:21.376804 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:21.377097 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 16:02:21.377275 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:02:21.378350 9560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:21.471911 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 9435
I20260929 16:02:21.489987 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 9571
Sep 29 16:02:21 dist-test-slave-cq88 krb5kdc[9193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 16:02:21.515749 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:21.490894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.23.222.67:32945)
0929 16:02:21.491071 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:21.491076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:21.491096 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:02:21.491228 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:21.491232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:21.491364 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:21.491450 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:21.492782 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.493563 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.494995 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:21.495531 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:21.495642 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:21.509955 (+ 14313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:21.509973 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:21.509990 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:21.510035 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:21.513042 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:21.513780 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:21.513786 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:21.513789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:21.513938 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:21.514267 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:21.514270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:21.514272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:21.514646 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:21.514867 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:21.515136 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:02:21.515365 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:02:21.516417 9695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:21.606559 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 9571
I20260929 16:02:21.624373 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 9206
I20260929 16:02:21.625583 9269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:21.739032 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 9206
2026-09-29T16:02:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4907 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T16:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:21Z Disabled control of system clock
I20260929 16:02:21.823565 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:33777
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:33777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:22.186038 9759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:22.186339 9759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:22.186496 9759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:22.194054 9759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:22.194176 9759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:22.194214 9759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:22.194268 9759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:22.194312 9759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:22.203859 9759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:33777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:33777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:22.205190 9759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:22.206884 9759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:22.219693 9764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:22.219674 9767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:22.219724 9765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:22.220582 9759 server_base.cc:1034] running on GCE node
I20260929 16:02:22.221426 9759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:22.223027 9759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:22.224303 9759 hybrid_clock.cc:648] HybridClock initialized: now 1790697742224248 us; error 50 us; skew 500 ppm
I20260929 16:02:22.224746 9759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:22.227566 9759 webserver.cc:460] Webserver started at http://127.23.222.126:33413/ using document root <none> and password file <none>
I20260929 16:02:22.228353 9759 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:22.228503 9759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:22.228878 9759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:22.232589 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "a875286095594243bb0d4e97c69b7ec2"
format_stamp: "Formatted at 2026-09-29 16:02:22 on dist-test-slave-cq88"
server_key: "588855b041a783ccf86f23f9e9ab750d"
server_key_iv: "b94bf5095b1db46f1dcd1df2af2fadfb"
server_key_version: "encryptionkey@0"
I20260929 16:02:22.233492 9759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "a875286095594243bb0d4e97c69b7ec2"
format_stamp: "Formatted at 2026-09-29 16:02:22 on dist-test-slave-cq88"
server_key: "588855b041a783ccf86f23f9e9ab750d"
server_key_iv: "b94bf5095b1db46f1dcd1df2af2fadfb"
server_key_version: "encryptionkey@0"
I20260929 16:02:22.239703 9759 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260929 16:02:22.243666 9773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:22.245148 9759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:22.245376 9759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "a875286095594243bb0d4e97c69b7ec2"
format_stamp: "Formatted at 2026-09-29 16:02:22 on dist-test-slave-cq88"
server_key: "588855b041a783ccf86f23f9e9ab750d"
server_key_iv: "b94bf5095b1db46f1dcd1df2af2fadfb"
server_key_version: "encryptionkey@0"
I20260929 16:02:22.245633 9759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:22.263553 9759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:22.264492 9759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:22.264814 9759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:22.285457 9759 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:33777
I20260929 16:02:22.285490 9824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:33777 every 8 connection(s)
I20260929 16:02:22.286967 9759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:22.290670 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 9759
I20260929 16:02:22.290887 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:22.291144 9825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:22.291460 24441 external_mini_cluster.cc:1428] Setting key 72a27f9a6b8da9e6d24509d3c3815f27
I20260929 16:02:22.305465 9825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2: Bootstrap starting.
I20260929 16:02:22.306314 9828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:22.293325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.23.222.126:33777)
0929 16:02:22.294492 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:22.294515 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:22.294574 (+ 59us) server_negotiation.cc:407] Connection header received
0929 16:02:22.296568 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:22.296589 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:22.297229 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:22.297713 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:22.298341 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:22.299774 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:22.301341 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:22.302136 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:22.302362 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:22.303844 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:22.303859 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:22.303881 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:22.304482 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:22.304562 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:22.304584 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:22.304720 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:22.304951 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 16:02:22.305283 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":383,"threads_started":1}
I20260929 16:02:22.308813 9825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:22.310011 9825 log.cc:824] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:22.313437 9825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2: No bootstrap required, opened a new log
I20260929 16:02:22.321075 9825 raft_consensus.cc:348] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a875286095594243bb0d4e97c69b7ec2" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33777 } }
I20260929 16:02:22.321516 9825 raft_consensus.cc:374] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:22.321630 9825 raft_consensus.cc:729] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a875286095594243bb0d4e97c69b7ec2, State: Initialized, Role: FOLLOWER
I20260929 16:02:22.322280 9825 consensus_queue.cc:260] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a875286095594243bb0d4e97c69b7ec2" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33777 } }
I20260929 16:02:22.322533 9825 raft_consensus.cc:388] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:22.322677 9825 raft_consensus.cc:482] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:22.322851 9825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:22.325414 9825 raft_consensus.cc:504] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a875286095594243bb0d4e97c69b7ec2" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33777 } }
I20260929 16:02:22.325947 9825 leader_election.cc:302] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a875286095594243bb0d4e97c69b7ec2; no voters:
I20260929 16:02:22.326565 9825 leader_election.cc:288] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:22.326754 9831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:22.327630 9831 raft_consensus.cc:686] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [term 1 LEADER]: Becoming Leader. State: Replica: a875286095594243bb0d4e97c69b7ec2, State: Running, Role: LEADER
I20260929 16:02:22.328186 9831 consensus_queue.cc:237] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a875286095594243bb0d4e97c69b7ec2" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33777 } }
I20260929 16:02:22.328845 9825 sys_catalog.cc:564] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:22.331171 9832 sys_catalog.cc:455] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a875286095594243bb0d4e97c69b7ec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a875286095594243bb0d4e97c69b7ec2" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33777 } } }
I20260929 16:02:22.331290 9833 sys_catalog.cc:455] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a875286095594243bb0d4e97c69b7ec2. Latest consensus state: current_term: 1 leader_uuid: "a875286095594243bb0d4e97c69b7ec2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a875286095594243bb0d4e97c69b7ec2" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 33777 } } }
I20260929 16:02:22.331636 9832 sys_catalog.cc:458] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:22.331699 9833 sys_catalog.cc:458] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:22.334800 9836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:22.340617 9836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:22.349344 9836 catalog_manager.cc:1347] Generated new cluster ID: adc95929b974437caae3ceb50b6a12c8
I20260929 16:02:22.349479 9836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:22.384678 9836 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:22.386018 9836 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:22.392416 9836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2: Generated new TSK 0
I20260929 16:02:22.393270 9836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:22.408330 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33777
--builtin_ntp_servers=127.23.222.84:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:22.772707 9850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:22.773062 9850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:22.773232 9850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:22.780792 9850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:22.780949 9850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:22.781139 9850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:02:22.790541 9850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:22.791929 9850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:22.793758 9850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:22.807312 9856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:22.807940 9855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:22.808663 9858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:22.809155 9850 server_base.cc:1034] running on GCE node
I20260929 16:02:22.809801 9850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:22.810838 9850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:22.812089 9850 hybrid_clock.cc:648] HybridClock initialized: now 1790697742812040 us; error 42 us; skew 500 ppm
I20260929 16:02:22.812433 9850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:22.814848 9850 webserver.cc:460] Webserver started at http://127.23.222.65:44827/ using document root <none> and password file <none>
I20260929 16:02:22.815704 9850 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:22.815845 9850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:22.816184 9850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:22.819387 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "8e3996b8564745c7bd370c2b4c94a141"
format_stamp: "Formatted at 2026-09-29 16:02:22 on dist-test-slave-cq88"
server_key: "f0359f4206df3f00acf9d91ed0697da9"
server_key_iv: "d9f4627affe4b3a07348d3d9c3c3a899"
server_key_version: "encryptionkey@0"
I20260929 16:02:22.820255 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "8e3996b8564745c7bd370c2b4c94a141"
format_stamp: "Formatted at 2026-09-29 16:02:22 on dist-test-slave-cq88"
server_key: "f0359f4206df3f00acf9d91ed0697da9"
server_key_iv: "d9f4627affe4b3a07348d3d9c3c3a899"
server_key_version: "encryptionkey@0"
I20260929 16:02:22.825765 9850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:02:22.829703 9864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:22.831466 9850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:22.831629 9850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "8e3996b8564745c7bd370c2b4c94a141"
format_stamp: "Formatted at 2026-09-29 16:02:22 on dist-test-slave-cq88"
server_key: "f0359f4206df3f00acf9d91ed0697da9"
server_key_iv: "d9f4627affe4b3a07348d3d9c3c3a899"
server_key_version: "encryptionkey@0"
I20260929 16:02:22.831831 9850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:22.846884 9850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:22.847991 9850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:22.848315 9850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:22.849452 9850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:22.851217 9850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:22.851398 9850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 16:02:22.851554 9850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:22.851665 9850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:22.887507 9850 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:46431
I20260929 16:02:22.887531 9977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:46431 every 8 connection(s)
I20260929 16:02:22.888950 9850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:22.898854 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 9850
I20260929 16:02:22.899052 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:22.899708 24441 external_mini_cluster.cc:1428] Setting key da1fb5682cf5152a86d3f334fa435783
I20260929 16:02:22.902249 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:22.891624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:59167 (local address 127.23.222.126:33777)
0929 16:02:22.892141 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:22.892148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:22.893487 (+ 1339us) server_negotiation.cc:407] Connection header received
0929 16:02:22.895393 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:22.895398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:22.895486 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:22.895590 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:22.896682 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:22.897380 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:22.898923 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:22.899617 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:22.899766 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:22.901322 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:22.901327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:22.901330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:22.901348 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:22.901388 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:22.901400 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:22.901473 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:22.901780 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 16:02:22.901898 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260929 16:02:22.902884 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33777
--builtin_ntp_servers=127.23.222.84:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:22.903375 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:22.892251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:33777 (local address 127.23.222.65:59167)
0929 16:02:22.893296 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:22.893347 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:22.895057 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:22.895809 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:22.895818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:22.895912 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:22.896524 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:22.896543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:22.897514 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:22.897517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:22.898767 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:22.898779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:22.899878 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:22.899884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:22.900099 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:22.900957 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:22.900960 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:22.901015 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:22.901120 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:22.901506 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:22.901518 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:22.901526 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:02:22.901756 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 16:02:22.902116 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":345,"threads_started":1}
I20260929 16:02:22.904934 9978 heartbeater.cc:347] Connected to a master server at 127.23.222.126:33777
I20260929 16:02:22.905236 9978 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:22.905994 9978 heartbeater.cc:511] Master 127.23.222.126:33777 requested a full tablet report, sending...
I20260929 16:02:22.908150 9790 ts_manager.cc:194] Registered new tserver with Master: 8e3996b8564745c7bd370c2b4c94a141 (127.23.222.65:46431)
I20260929 16:02:22.909595 9790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.65:59167
W20260929 16:02:23.235775 9982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:23.236163 9982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:23.236392 9982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:23.243957 9982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:23.244102 9982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:23.244285 9982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:02:23.253707 9982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:23.255074 9982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:23.256983 9982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:23.270018 9990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:23.270081 9988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:23.270366 9987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:23.271389 9982 server_base.cc:1034] running on GCE node
I20260929 16:02:23.272040 9982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:23.273047 9982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:23.274299 9982 hybrid_clock.cc:648] HybridClock initialized: now 1790697743274244 us; error 65 us; skew 500 ppm
I20260929 16:02:23.274652 9982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:23.277072 9982 webserver.cc:460] Webserver started at http://127.23.222.66:45071/ using document root <none> and password file <none>
I20260929 16:02:23.277798 9982 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:23.277942 9982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:23.278267 9982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:23.281277 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "681b23a15d774d73b1c6b461437e66cc"
format_stamp: "Formatted at 2026-09-29 16:02:23 on dist-test-slave-cq88"
server_key: "4b383c036c9dc5558c8dd810c7877c61"
server_key_iv: "58b0769b1c60cfccde468f8c2d10ca03"
server_key_version: "encryptionkey@0"
I20260929 16:02:23.282089 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "681b23a15d774d73b1c6b461437e66cc"
format_stamp: "Formatted at 2026-09-29 16:02:23 on dist-test-slave-cq88"
server_key: "4b383c036c9dc5558c8dd810c7877c61"
server_key_iv: "58b0769b1c60cfccde468f8c2d10ca03"
server_key_version: "encryptionkey@0"
I20260929 16:02:23.287707 9982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260929 16:02:23.291992 9996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:23.293922 9982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 16:02:23.294117 9982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "681b23a15d774d73b1c6b461437e66cc"
format_stamp: "Formatted at 2026-09-29 16:02:23 on dist-test-slave-cq88"
server_key: "4b383c036c9dc5558c8dd810c7877c61"
server_key_iv: "58b0769b1c60cfccde468f8c2d10ca03"
server_key_version: "encryptionkey@0"
I20260929 16:02:23.294353 9982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:23.313978 9982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:23.315146 9982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:23.315536 9982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:23.316694 9982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:23.318311 9982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:23.318454 9982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:23.318588 9982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:23.318683 9982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:23.354027 9982 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:44105
I20260929 16:02:23.354059 10109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:44105 every 8 connection(s)
I20260929 16:02:23.355518 9982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:23.358881 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 9982
I20260929 16:02:23.359192 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:23.359876 24441 external_mini_cluster.cc:1428] Setting key 6112162946b7ef7fa6a7f23aedad564b
I20260929 16:02:23.362928 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33777
--builtin_ntp_servers=127.23.222.84:36075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:23.370184 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.358428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:54695 (local address 127.23.222.126:33777)
0929 16:02:23.358691 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:23.358700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:23.360260 (+ 1560us) server_negotiation.cc:407] Connection header received
0929 16:02:23.362077 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:23.362083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:23.362170 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:23.362285 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:23.363686 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.364949 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.366646 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.367405 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.367569 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.369137 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:23.369146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:23.369154 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:23.369183 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:23.369229 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:23.369248 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:23.369346 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:23.369660 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 16:02:23.369786 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:02:23.371280 10112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.358888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:33777 (local address 127.23.222.66:54695)
0929 16:02:23.360062 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:23.360118 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:23.361795 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:23.362557 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:23.362566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:23.362667 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:23.363486 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.363507 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.365138 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.365143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.366471 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.366484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.367676 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.367681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.367908 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.368738 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:23.368762 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:23.368808 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:23.368903 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:23.369384 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:23.369394 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:23.369401 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 16:02:23.369615 (+ 214us) client_negotiation.cc:240] Negotiation successful
0929 16:02:23.370000 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":436,"threads_started":1}
I20260929 16:02:23.372897 10110 heartbeater.cc:347] Connected to a master server at 127.23.222.126:33777
I20260929 16:02:23.373256 10110 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:23.374212 10110 heartbeater.cc:511] Master 127.23.222.126:33777 requested a full tablet report, sending...
I20260929 16:02:23.376073 9790 ts_manager.cc:194] Registered new tserver with Master: 681b23a15d774d73b1c6b461437e66cc (127.23.222.66:44105)
I20260929 16:02:23.376940 9790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.66:54695
W20260929 16:02:23.717451 10113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:23.717851 10113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:23.718025 10113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:23.725673 10113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:23.725841 10113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:23.726029 10113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:02:23.735961 10113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:36075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:33777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:23.737428 10113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:23.739339 10113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:23.754112 10118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:23.754184 10121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:23.754150 10119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:23.754969 10113 server_base.cc:1034] running on GCE node
I20260929 16:02:23.755749 10113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:23.756940 10113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:23.758179 10113 hybrid_clock.cc:648] HybridClock initialized: now 1790697743758113 us; error 48 us; skew 500 ppm
I20260929 16:02:23.758527 10113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:23.761044 10113 webserver.cc:460] Webserver started at http://127.23.222.67:37001/ using document root <none> and password file <none>
I20260929 16:02:23.761780 10113 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:23.761871 10113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:23.762214 10113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:23.765420 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "43b222846a9b4feead1e57e316fbad40"
format_stamp: "Formatted at 2026-09-29 16:02:23 on dist-test-slave-cq88"
server_key: "0c73790e1864ad88d265a89cfe4bdb33"
server_key_iv: "46680a9d8078ac49b391bf980edad2a8"
server_key_version: "encryptionkey@0"
I20260929 16:02:23.766250 10113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "43b222846a9b4feead1e57e316fbad40"
format_stamp: "Formatted at 2026-09-29 16:02:23 on dist-test-slave-cq88"
server_key: "0c73790e1864ad88d265a89cfe4bdb33"
server_key_iv: "46680a9d8078ac49b391bf980edad2a8"
server_key_version: "encryptionkey@0"
I20260929 16:02:23.771709 10113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:02:23.775205 10127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:23.776580 10113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:23.776747 10113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "43b222846a9b4feead1e57e316fbad40"
format_stamp: "Formatted at 2026-09-29 16:02:23 on dist-test-slave-cq88"
server_key: "0c73790e1864ad88d265a89cfe4bdb33"
server_key_iv: "46680a9d8078ac49b391bf980edad2a8"
server_key_version: "encryptionkey@0"
I20260929 16:02:23.776969 10113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:23.789716 10113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:23.790536 10113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:23.790835 10113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:23.791914 10113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:23.793350 10113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:23.793459 10113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:23.793589 10113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:23.793675 10113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:23.827001 10113 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:39833
I20260929 16:02:23.827057 10241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:39833 every 8 connection(s)
I20260929 16:02:23.828418 10113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:23.832783 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 10113
I20260929 16:02:23.833086 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:23.833551 24441 external_mini_cluster.cc:1428] Setting key 26595324324e87a2f84f82b6d461f119
I20260929 16:02:23.841441 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.831206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:45061 (local address 127.23.222.126:33777)
0929 16:02:23.831406 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:23.831413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:23.832801 (+ 1388us) server_negotiation.cc:407] Connection header received
0929 16:02:23.835068 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:23.835074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:23.835172 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:23.835303 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:23.836482 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.837240 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.838744 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.839146 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.839292 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.840637 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:23.840642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:23.840645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:23.840662 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:23.840692 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:23.840704 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:23.840801 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:23.841013 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 16:02:23.841111 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:02:23.842388 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.831795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:33777 (local address 127.23.222.67:45061)
0929 16:02:23.832653 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:23.832691 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:23.834765 (+ 2074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:23.835551 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:23.835560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:23.835708 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:23.836313 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.836342 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.837390 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.837395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.838605 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.838616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.839391 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.839398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.839636 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.840323 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:23.840325 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:23.840363 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:23.840442 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:23.840797 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:23.840808 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:23.840819 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:02:23.841002 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 16:02:23.841372 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":265,"threads_started":1}
I20260929 16:02:23.843683 10242 heartbeater.cc:347] Connected to a master server at 127.23.222.126:33777
I20260929 16:02:23.843932 10242 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:23.844595 10242 heartbeater.cc:511] Master 127.23.222.126:33777 requested a full tablet report, sending...
I20260929 16:02:23.845991 9790 ts_manager.cc:194] Registered new tserver with Master: 43b222846a9b4feead1e57e316fbad40 (127.23.222.67:39833)
I20260929 16:02:23.847112 9790 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.67:45061
I20260929 16:02:23.849004 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:23.860159 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.853467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.23.222.126:33777)
0929 16:02:23.853645 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:23.853650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:23.854188 (+ 538us) server_negotiation.cc:407] Connection header received
0929 16:02:23.854353 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:23.854357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:23.854435 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:23.854520 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:23.855182 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.855969 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.857632 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.858144 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.858260 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.859491 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:23.859496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:23.859499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:23.859519 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:23.859553 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:23.859565 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:23.859640 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:23.859844 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:02:23.859924 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:02:23.864133 9789 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:23.871949 9789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:23.906697 10258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.884982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:44105 (local address 127.0.0.1:51328)
0929 16:02:23.890169 (+ 5187us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:23.890190 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:23.890325 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:23.898554 (+ 8229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:23.898568 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:23.898795 (+ 227us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:23.899523 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.899541 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.901469 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.901477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.903159 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.903173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.905601 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.905608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.905871 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.905940 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 16:02:23.906067 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 16:02:23.906279 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5034,"mutex_wait_us":28,"thread_start_us":13926,"threads_started":1}
I20260929 16:02:23.906826 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.885502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.23.222.66:44105)
0929 16:02:23.897584 (+ 12082us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:23.897595 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:23.897638 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:02:23.897786 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:23.897798 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:23.898162 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:23.898477 (+ 315us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:23.899756 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.901327 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.903456 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.905035 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.905168 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.905460 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:23.906273 (+ 813us) server_negotiation.cc:299] Negotiation successful
0929 16:02:23.906478 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11845,"thread_start_us":11740,"threads_started":1}
I20260929 16:02:23.907850 10255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.884975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.0.0.1:52596)
0929 16:02:23.889605 (+ 4630us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:23.889654 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:23.889881 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:23.900365 (+ 10484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:23.900369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:23.900409 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:23.900855 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.900867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.902863 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.902868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.904590 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.904603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.906886 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.906895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.907173 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.907206 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:02:23.907332 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 16:02:23.907486 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4310,"mutex_wait_us":2,"thread_start_us":4129,"threads_started":1}
I20260929 16:02:23.908185 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.885361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.23.222.65:46431)
0929 16:02:23.899505 (+ 14144us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:23.899513 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:23.899545 (+ 32us) server_negotiation.cc:407] Connection header received
0929 16:02:23.899648 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:23.899657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:23.899914 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:23.900119 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:23.901042 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.902699 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.904873 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.906338 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.906478 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.906752 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:23.907629 (+ 877us) server_negotiation.cc:299] Negotiation successful
0929 16:02:23.907823 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13941,"thread_start_us":13787,"threads_started":1}
I20260929 16:02:23.908507 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.887792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.23.222.67:39833)
0929 16:02:23.897023 (+ 9231us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:23.897039 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:23.899588 (+ 2549us) server_negotiation.cc:407] Connection header received
0929 16:02:23.899914 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:23.899922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:23.900132 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:23.900286 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:23.901316 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.902861 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.905605 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.907077 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.907214 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.907497 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:23.907961 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 16:02:23.908139 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8991,"thread_start_us":8792,"threads_started":1}
I20260929 16:02:23.908808 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:23.887196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:39833 (local address 127.0.0.1:33940)
0929 16:02:23.899438 (+ 12242us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:23.899463 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:23.899705 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:23.900614 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:23.900621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:23.900648 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:23.901108 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.901119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.903013 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.903019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.905362 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:23.905376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:23.907293 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:23.907303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:23.907473 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:23.907506 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:02:23.907588 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 16:02:23.907713 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12098,"mutex_wait_us":26,"thread_start_us":2679,"threads_started":1}
I20260929 16:02:23.911944 9912 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7b7606b0464239b8fad9ca27055c46 (DEFAULT_TABLE table=test-table [id=e18ac0c453b84110b0f151ea507335a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:23.911943 10045 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7b7606b0464239b8fad9ca27055c46 (DEFAULT_TABLE table=test-table [id=e18ac0c453b84110b0f151ea507335a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:23.912736 10176 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7b7606b0464239b8fad9ca27055c46 (DEFAULT_TABLE table=test-table [id=e18ac0c453b84110b0f151ea507335a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:23.913228 9978 heartbeater.cc:503] Master 127.23.222.126:33777 was elected leader, sending a full tablet report...
I20260929 16:02:23.913620 10045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7b7606b0464239b8fad9ca27055c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:23.913839 9912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7b7606b0464239b8fad9ca27055c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:23.914215 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7b7606b0464239b8fad9ca27055c46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:23.927588 10260 tablet_bootstrap.cc:492] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: Bootstrap starting.
I20260929 16:02:23.929863 10261 tablet_bootstrap.cc:492] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141: Bootstrap starting.
I20260929 16:02:23.931703 10260 tablet_bootstrap.cc:654] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:23.932183 10262 tablet_bootstrap.cc:492] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: Bootstrap starting.
I20260929 16:02:23.933136 10260 log.cc:824] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:23.934231 10261 tablet_bootstrap.cc:654] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:23.935667 10262 tablet_bootstrap.cc:654] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:23.935851 10261 log.cc:824] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:23.936893 10262 log.cc:824] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:23.938923 10260 tablet_bootstrap.cc:492] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: No bootstrap required, opened a new log
I20260929 16:02:23.939333 10260 ts_tablet_manager.cc:1397] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260929 16:02:23.939589 10262 tablet_bootstrap.cc:492] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: No bootstrap required, opened a new log
I20260929 16:02:23.939842 10262 ts_tablet_manager.cc:1397] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 16:02:23.946322 10261 tablet_bootstrap.cc:492] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141: No bootstrap required, opened a new log
I20260929 16:02:23.946640 10261 ts_tablet_manager.cc:1397] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260929 16:02:23.947516 10262 raft_consensus.cc:348] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } }
I20260929 16:02:23.948102 10262 raft_consensus.cc:374] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:23.948220 10262 raft_consensus.cc:729] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681b23a15d774d73b1c6b461437e66cc, State: Initialized, Role: FOLLOWER
I20260929 16:02:23.948844 10262 consensus_queue.cc:260] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } }
I20260929 16:02:23.950093 10110 heartbeater.cc:503] Master 127.23.222.126:33777 was elected leader, sending a full tablet report...
I20260929 16:02:23.950909 10262 ts_tablet_manager.cc:1428] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260929 16:02:23.950887 10260 raft_consensus.cc:348] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } }
I20260929 16:02:23.951392 10260 raft_consensus.cc:374] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:23.951488 10260 raft_consensus.cc:729] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43b222846a9b4feead1e57e316fbad40, State: Initialized, Role: FOLLOWER
I20260929 16:02:23.952102 10260 consensus_queue.cc:260] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } }
I20260929 16:02:23.956769 10242 heartbeater.cc:503] Master 127.23.222.126:33777 was elected leader, sending a full tablet report...
I20260929 16:02:23.957508 10260 ts_tablet_manager.cc:1428] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: Time spent starting tablet: real 0.018s user 0.017s sys 0.000s
I20260929 16:02:23.960491 10261 raft_consensus.cc:348] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } }
I20260929 16:02:23.961025 10261 raft_consensus.cc:374] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:23.961134 10261 raft_consensus.cc:729] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e3996b8564745c7bd370c2b4c94a141, State: Initialized, Role: FOLLOWER
I20260929 16:02:23.961822 10261 consensus_queue.cc:260] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } }
I20260929 16:02:23.963517 10261 ts_tablet_manager.cc:1428] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260929 16:02:24.080336 10243 tablet.cc:2369] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:24.107782 10111 tablet.cc:2369] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:24.135869 10267 raft_consensus.cc:482] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:24.136230 10267 raft_consensus.cc:504] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } }
I20260929 16:02:24.137919 10267 leader_election.cc:288] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 681b23a15d774d73b1c6b461437e66cc (127.23.222.66:44105), 8e3996b8564745c7bd370c2b4c94a141 (127.23.222.65:46431)
W20260929 16:02:24.141711 9979 tablet.cc:2369] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:24.144706 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.138443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:50243 (local address 127.23.222.66:44105)
0929 16:02:24.138651 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:24.138657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:24.138920 (+ 263us) server_negotiation.cc:407] Connection header received
0929 16:02:24.139165 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:24.139170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:24.139297 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:24.139395 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:24.139940 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.140793 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.142743 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.143899 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.144014 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:24.144063 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:24.144403 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 16:02:24.144508 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:02:24.144848 10244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.138428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:44105 (local address 127.23.222.67:50243)
0929 16:02:24.138779 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.138810 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:24.139005 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:24.139549 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:24.139553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:24.139581 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:24.139777 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:24.139789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.140953 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.140958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:24.142588 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:24.142600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.144057 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.144063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:24.144238 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:24.144275 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:02:24.144383 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:02:24.144528 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206}
I20260929 16:02:24.145776 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.138500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:51965 (local address 127.23.222.65:46431)
0929 16:02:24.138693 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:24.138699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:24.139490 (+ 791us) server_negotiation.cc:407] Connection header received
0929 16:02:24.139571 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:24.139576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:24.139665 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:24.139763 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:24.140380 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.141254 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.143763 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.144997 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.145161 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:24.145261 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:24.145474 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 16:02:24.145611 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:02:24.145845 10269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.138438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.23.222.67:51965)
0929 16:02:24.139213 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.139251 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:24.139392 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:24.139783 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:24.139786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:24.139805 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:24.140201 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:24.140214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.141435 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.141439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:24.143533 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:24.143549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.145177 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.145185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:24.145326 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:24.145348 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 16:02:24.145418 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 16:02:24.145499 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":33,"thread_start_us":429,"threads_started":1}
I20260929 16:02:24.146106 10065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b7606b0464239b8fad9ca27055c46" candidate_uuid: "43b222846a9b4feead1e57e316fbad40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681b23a15d774d73b1c6b461437e66cc" is_pre_election: true
I20260929 16:02:24.146661 10065 raft_consensus.cc:2454] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43b222846a9b4feead1e57e316fbad40 in term 0.
I20260929 16:02:24.146883 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b7606b0464239b8fad9ca27055c46" candidate_uuid: "43b222846a9b4feead1e57e316fbad40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3996b8564745c7bd370c2b4c94a141" is_pre_election: true
I20260929 16:02:24.147435 10132 leader_election.cc:302] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b222846a9b4feead1e57e316fbad40, 681b23a15d774d73b1c6b461437e66cc; no voters:
I20260929 16:02:24.147459 9932 raft_consensus.cc:2454] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43b222846a9b4feead1e57e316fbad40 in term 0.
I20260929 16:02:24.147828 10267 raft_consensus.cc:2781] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:24.148000 10267 raft_consensus.cc:482] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:24.148133 10267 raft_consensus.cc:3037] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:24.150502 10267 raft_consensus.cc:504] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } }
I20260929 16:02:24.151161 10267 leader_election.cc:288] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [CANDIDATE]: Term 1 election: Requested vote from peers 681b23a15d774d73b1c6b461437e66cc (127.23.222.66:44105), 8e3996b8564745c7bd370c2b4c94a141 (127.23.222.65:46431)
I20260929 16:02:24.151659 10065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b7606b0464239b8fad9ca27055c46" candidate_uuid: "43b222846a9b4feead1e57e316fbad40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681b23a15d774d73b1c6b461437e66cc"
I20260929 16:02:24.151695 9932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b7b7606b0464239b8fad9ca27055c46" candidate_uuid: "43b222846a9b4feead1e57e316fbad40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e3996b8564745c7bd370c2b4c94a141"
I20260929 16:02:24.151852 10065 raft_consensus.cc:3037] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:24.151988 9932 raft_consensus.cc:3037] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:24.154106 10065 raft_consensus.cc:2454] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43b222846a9b4feead1e57e316fbad40 in term 1.
I20260929 16:02:24.154162 9932 raft_consensus.cc:2454] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43b222846a9b4feead1e57e316fbad40 in term 1.
I20260929 16:02:24.154554 10132 leader_election.cc:302] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43b222846a9b4feead1e57e316fbad40, 681b23a15d774d73b1c6b461437e66cc; no voters:
I20260929 16:02:24.154912 10267 raft_consensus.cc:2781] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:24.155403 10267 raft_consensus.cc:686] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 1 LEADER]: Becoming Leader. State: Replica: 43b222846a9b4feead1e57e316fbad40, State: Running, Role: LEADER
I20260929 16:02:24.155948 10267 consensus_queue.cc:237] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } }
I20260929 16:02:24.160384 9789 catalog_manager.cc:5495] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43b222846a9b4feead1e57e316fbad40 (127.23.222.67). New cstate: current_term: 1 leader_uuid: "43b222846a9b4feead1e57e316fbad40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43b222846a9b4feead1e57e316fbad40" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39833 } health_report { overall_health: HEALTHY } } }
I20260929 16:02:24.198413 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.192361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33942 (local address 127.23.222.67:39833)
0929 16:02:24.192577 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:24.192584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:24.192687 (+ 103us) server_negotiation.cc:407] Connection header received
0929 16:02:24.192955 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:24.192960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:24.193041 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:24.193203 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:24.193572 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.194316 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.195902 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.196308 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.196457 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:24.196988 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:24.197513 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:24.197890 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:24.198002 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 16:02:24.198092 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:02:24.207430 10065 raft_consensus.cc:1261] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [term 1 FOLLOWER]: Refusing update from remote peer 43b222846a9b4feead1e57e316fbad40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:24.207425 9932 raft_consensus.cc:1261] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [term 1 FOLLOWER]: Refusing update from remote peer 43b222846a9b4feead1e57e316fbad40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:24.208551 10270 consensus_queue.cc:1035] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e3996b8564745c7bd370c2b4c94a141" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 46431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:24.208999 10267 consensus_queue.cc:1035] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681b23a15d774d73b1c6b461437e66cc" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 44105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:24.221539 10273 mvcc.cc:205] Tried to move back new op lower bound from 7334697960261922816 to 7334697960066854912. Current Snapshot: MvccSnapshot[applied={T|T < 7334697960261922816}]
I20260929 16:02:24.236864 9789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:24.237136 9789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:24.241106 9789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a875286095594243bb0d4e97c69b7ec2: Sending DeleteTablet for 3 replicas of tablet 8b7b7606b0464239b8fad9ca27055c46
I20260929 16:02:24.242376 10045 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7b7606b0464239b8fad9ca27055c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:24 UTC) from {username='slave'} at 127.0.0.1:51328
I20260929 16:02:24.242429 9912 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7b7606b0464239b8fad9ca27055c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:24 UTC) from {username='slave'} at 127.0.0.1:52596
I20260929 16:02:24.242843 10176 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b7b7606b0464239b8fad9ca27055c46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:24 UTC) from {username='slave'} at 127.0.0.1:33940
I20260929 16:02:24.244944 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 9850
I20260929 16:02:24.271663 10285 tablet_replica.cc:330] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: stopping tablet replica
I20260929 16:02:24.274016 10285 raft_consensus.cc:2229] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:24.280663 10284 tablet_replica.cc:330] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: stopping tablet replica
I20260929 16:02:24.281263 10284 raft_consensus.cc:2229] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:24.279814 10285 raft_consensus.cc:2258] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:24.282363 10284 raft_consensus.cc:2258] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:24.283226 10283 tablet_replica.cc:330] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141: stopping tablet replica
I20260929 16:02:24.283679 10283 raft_consensus.cc:2229] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:24.284168 10283 raft_consensus.cc:2258] T 8b7b7606b0464239b8fad9ca27055c46 P 8e3996b8564745c7bd370c2b4c94a141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:24.284294 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.246554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52612 (local address 127.23.222.65:46431)
0929 16:02:24.246789 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:24.246798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:24.270666 (+ 23868us) server_negotiation.cc:407] Connection header received
0929 16:02:24.272279 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:24.272290 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:24.272445 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:24.272560 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:24.274399 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.276092 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.280024 (+ 3932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.280642 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.280826 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:24.282427 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:24.282443 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:24.282461 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:24.283349 (+ 888us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:24.283442 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:24.283469 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:24.283602 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:24.283763 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 16:02:24.283896 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:02:24.285318 9973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:24.285624 10285 ts_tablet_manager.cc:1905] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:24.290068 10284 ts_tablet_manager.cc:1905] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:24.298020 10285 ts_tablet_manager.cc:1918] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:24.298357 10285 log.cc:1196] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/8b7b7606b0464239b8fad9ca27055c46
I20260929 16:02:24.299145 10285 ts_tablet_manager.cc:1939] T 8b7b7606b0464239b8fad9ca27055c46 P 43b222846a9b4feead1e57e316fbad40: Deleting consensus metadata
I20260929 16:02:24.302268 9776 catalog_manager.cc:4841] TS 43b222846a9b4feead1e57e316fbad40 (127.23.222.67:39833): tablet 8b7b7606b0464239b8fad9ca27055c46 (table test-table [id=e18ac0c453b84110b0f151ea507335a4]) successfully deleted
I20260929 16:02:24.304715 10284 ts_tablet_manager.cc:1918] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:24.304946 10284 log.cc:1196] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/8b7b7606b0464239b8fad9ca27055c46
I20260929 16:02:24.305573 10284 ts_tablet_manager.cc:1939] T 8b7b7606b0464239b8fad9ca27055c46 P 681b23a15d774d73b1c6b461437e66cc: Deleting consensus metadata
I20260929 16:02:24.307623 9777 catalog_manager.cc:4841] TS 681b23a15d774d73b1c6b461437e66cc (127.23.222.66:44105): tablet 8b7b7606b0464239b8fad9ca27055c46 (table test-table [id=e18ac0c453b84110b0f151ea507335a4]) successfully deleted
I20260929 16:02:24.397981 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 9850
W20260929 16:02:24.416756 9775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.222.65:46431 (error 108)
I20260929 16:02:24.416961 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 9982
I20260929 16:02:24.424113 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.420346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.0.0.1:52616)
0929 16:02:24.423712 (+ 3366us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.423868 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3214,"thread_start_us":3113,"threads_started":1}
W20260929 16:02:24.424461 9775 catalog_manager.cc:4584] TS 8e3996b8564745c7bd370c2b4c94a141 (127.23.222.65:46431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b7b7606b0464239b8fad9ca27055c46: Network error: Client connection negotiation failed: client connection to 127.23.222.65:46431: connect: Connection refused (error 111)
I20260929 16:02:24.427438 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.418786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.23.222.66:44105)
0929 16:02:24.419009 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:24.419015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:24.419039 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:02:24.419114 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:24.419120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:24.419230 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:24.419376 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:24.419957 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.421406 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.423904 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.424445 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.424594 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:24.425859 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:24.425869 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:24.425883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:24.426529 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:24.426583 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:24.426605 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:24.426744 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:24.426882 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 16:02:24.427017 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:02:24.428148 10105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:24.478240 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.477777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.0.0.1:52630)
0929 16:02:24.477990 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.478097 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 16:02:24.479454 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.479007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.0.0.1:52634)
0929 16:02:24.479182 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.479303 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 16:02:24.540295 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.539794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.0.0.1:52640)
0929 16:02:24.540029 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.540122 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 16:02:24.541517 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.541100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.0.0.1:52644)
0929 16:02:24.541281 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.541367 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 16:02:24.579790 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 9982
I20260929 16:02:24.598942 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 10113
I20260929 16:02:24.606527 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.600059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33958 (local address 127.23.222.67:39833)
0929 16:02:24.600278 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:24.600282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:24.600298 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:02:24.600353 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:24.600358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:24.600477 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:24.600595 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:24.601164 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.601962 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.603336 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:24.603785 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:24.603937 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:24.605061 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:24.605070 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:24.605085 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:24.605689 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:24.605740 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:24.605766 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:24.605895 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:24.606033 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 16:02:24.606150 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 16:02:24.607326 10237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:24.622949 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.622251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.0.0.1:52658)
0929 16:02:24.622577 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.622712 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 16:02:24.624339 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.623821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.0.0.1:52674)
0929 16:02:24.624042 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.624145 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260929 16:02:24.735194 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 10113
I20260929 16:02:24.756341 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 9759
I20260929 16:02:24.757619 9820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:24.866971 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:24.866373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:46431 (local address 127.0.0.1:52682)
0929 16:02:24.866654 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:24.866775 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:46431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 16:02:24.867156 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 9759
2026-09-29T16:02:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3119 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:24 dist-test-slave-cq88 krb5kdc[10295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:24 dist-test-slave-cq88 krb5kdc[10295](info): set up 2 sockets
Sep 29 16:02:24 dist-test-slave-cq88 krb5kdc[10295](info): commencing operation
krb5kdc: starting...
W20260929 16:02:25.982767 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:26 dist-test-slave-cq88 krb5kdc[10295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:02:26.268944 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38367
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:38367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:26.617859 10308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:26.618185 10308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:26.618350 10308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:26.625979 10308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:26.626106 10308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:26.626184 10308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:26.626271 10308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:26.626353 10308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:26.636873 10308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:38367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:26.638322 10308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:26.639825 10308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:26.652326 10313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:26.652273 10314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:26.652350 10316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:26.653754 10308 server_base.cc:1034] running on GCE node
I20260929 16:02:26.654682 10308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:26.656332 10308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:26.657554 10308 hybrid_clock.cc:648] HybridClock initialized: now 1790697746657507 us; error 35 us; skew 500 ppm
I20260929 16:02:26.658043 10308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:26 dist-test-slave-cq88 krb5kdc[10295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697746, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:26.671437 10308 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:02:26.673280 10308 webserver.cc:460] Webserver started at http://127.23.222.126:43647/ using document root <none> and password file <none>
I20260929 16:02:26.674103 10308 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:26.674269 10308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:26.674654 10308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:26.677991 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "f284506ede6a46e8b4905f421d114d2e"
format_stamp: "Formatted at 2026-09-29 16:02:26 on dist-test-slave-cq88"
server_key: "419b87ed5132daacd41658307d745f19"
server_key_iv: "cd37452c04fd865b3c54dcf616d3c4ec"
server_key_version: "encryptionkey@0"
I20260929 16:02:26.678817 10308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "f284506ede6a46e8b4905f421d114d2e"
format_stamp: "Formatted at 2026-09-29 16:02:26 on dist-test-slave-cq88"
server_key: "419b87ed5132daacd41658307d745f19"
server_key_iv: "cd37452c04fd865b3c54dcf616d3c4ec"
server_key_version: "encryptionkey@0"
I20260929 16:02:26.684474 10308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 16:02:26.688257 10323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:26.689764 10308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:26.689994 10308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "f284506ede6a46e8b4905f421d114d2e"
format_stamp: "Formatted at 2026-09-29 16:02:26 on dist-test-slave-cq88"
server_key: "419b87ed5132daacd41658307d745f19"
server_key_iv: "cd37452c04fd865b3c54dcf616d3c4ec"
server_key_version: "encryptionkey@0"
I20260929 16:02:26.690224 10308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:26.721119 10308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:26.726303 10308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:26.726655 10308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:26.747102 10308 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:38367
I20260929 16:02:26.747124 10375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:38367 every 8 connection(s)
I20260929 16:02:26.748629 10308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:26.752676 10376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:26.756265 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 10308
I20260929 16:02:26.756496 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:26.757037 24441 external_mini_cluster.cc:1428] Setting key 6bb1adc77b18f086fe3c721a575e7533
I20260929 16:02:26.766491 10376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e: Bootstrap starting.
I20260929 16:02:26.771636 10376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:26.773447 10376 log.cc:824] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:02:26 dist-test-slave-cq88 krb5kdc[10295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:26.777678 10376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e: No bootstrap required, opened a new log
I20260929 16:02:26.784953 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:26.759165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.23.222.126:38367)
0929 16:02:26.760076 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:26.760089 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:26.760136 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:02:26.761905 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:26.761929 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:26.762379 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:26.762753 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:26.764774 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:26.766295 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:26.768735 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:26.769560 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:26.769718 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:26.776123 (+ 6405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:26.776133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:26.776151 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:26.776193 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:26.779775 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:26.780874 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:26.780885 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:26.780892 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:26.781126 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:26.781648 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:26.781652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:26.781655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:26.782271 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:26.782553 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:26.783032 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 16:02:26.783724 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":394,"threads_started":1}
I20260929 16:02:26.787657 10376 raft_consensus.cc:348] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f284506ede6a46e8b4905f421d114d2e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38367 } }
I20260929 16:02:26.788082 10376 raft_consensus.cc:374] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:26.788211 10376 raft_consensus.cc:729] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f284506ede6a46e8b4905f421d114d2e, State: Initialized, Role: FOLLOWER
I20260929 16:02:26.788904 10376 consensus_queue.cc:260] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f284506ede6a46e8b4905f421d114d2e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38367 } }
I20260929 16:02:26.789157 10376 raft_consensus.cc:388] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:26.789260 10376 raft_consensus.cc:482] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:26.789393 10376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:26.791706 10376 raft_consensus.cc:504] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f284506ede6a46e8b4905f421d114d2e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38367 } }
I20260929 16:02:26.792336 10376 leader_election.cc:302] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f284506ede6a46e8b4905f421d114d2e; no voters:
I20260929 16:02:26.792948 10376 leader_election.cc:288] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:26.793131 10381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:26.793790 10381 raft_consensus.cc:686] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [term 1 LEADER]: Becoming Leader. State: Replica: f284506ede6a46e8b4905f421d114d2e, State: Running, Role: LEADER
I20260929 16:02:26.794449 10381 consensus_queue.cc:237] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f284506ede6a46e8b4905f421d114d2e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38367 } }
I20260929 16:02:26.796263 10376 sys_catalog.cc:564] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:26.797745 10383 sys_catalog.cc:455] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f284506ede6a46e8b4905f421d114d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f284506ede6a46e8b4905f421d114d2e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38367 } } }
I20260929 16:02:26.797753 10382 sys_catalog.cc:455] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f284506ede6a46e8b4905f421d114d2e. Latest consensus state: current_term: 1 leader_uuid: "f284506ede6a46e8b4905f421d114d2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f284506ede6a46e8b4905f421d114d2e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38367 } } }
I20260929 16:02:26.798243 10383 sys_catalog.cc:458] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:26.798243 10382 sys_catalog.cc:458] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:26.801260 10386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:26.808306 10386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:26.820327 10386 catalog_manager.cc:1347] Generated new cluster ID: 4e24310d608942b09f165b400fb8f0ac
I20260929 16:02:26.820478 10386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:26.834913 10386 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:26.836470 10386 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:26.848727 10386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e: Generated new TSK 0
I20260929 16:02:26.849910 10386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 16:02:26.914564 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38367
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:27.281543 10402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:27.281834 10402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:27.281997 10402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:27.290654 10402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:27.290773 10402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:27.290957 10402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:02:27.302054 10402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:27.303519 10402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:27.304858 10402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:27.318620 10407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:27.318576 10410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:27.318665 10408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:27.319799 10402 server_base.cc:1034] running on GCE node
I20260929 16:02:27.320484 10402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:27.321456 10402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:27.322659 10402 hybrid_clock.cc:648] HybridClock initialized: now 1790697747322583 us; error 70 us; skew 500 ppm
I20260929 16:02:27.322958 10402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:27 dist-test-slave-cq88 krb5kdc[10295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:27.326226 10402 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:02:27.327777 10402 webserver.cc:460] Webserver started at http://127.23.222.65:41767/ using document root <none> and password file <none>
I20260929 16:02:27.328464 10402 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:27.328567 10402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:27.328837 10402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:27.331995 10402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "f4dabd2cc0db4ceb9a254844180dc686"
format_stamp: "Formatted at 2026-09-29 16:02:27 on dist-test-slave-cq88"
server_key: "8023e31e564d425fb5ef666774308664"
server_key_iv: "442fd4a6dcd18bbb5462873af7c80e61"
server_key_version: "encryptionkey@0"
I20260929 16:02:27.332767 10402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "f4dabd2cc0db4ceb9a254844180dc686"
format_stamp: "Formatted at 2026-09-29 16:02:27 on dist-test-slave-cq88"
server_key: "8023e31e564d425fb5ef666774308664"
server_key_iv: "442fd4a6dcd18bbb5462873af7c80e61"
server_key_version: "encryptionkey@0"
I20260929 16:02:27.338768 10402 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 16:02:27.342684 10417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:27.344285 10402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:27.344455 10402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "f4dabd2cc0db4ceb9a254844180dc686"
format_stamp: "Formatted at 2026-09-29 16:02:27 on dist-test-slave-cq88"
server_key: "8023e31e564d425fb5ef666774308664"
server_key_iv: "442fd4a6dcd18bbb5462873af7c80e61"
server_key_version: "encryptionkey@0"
I20260929 16:02:27.344648 10402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:27.360266 10402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:27.364826 10402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:27.365095 10402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:27.366122 10402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:27.367820 10402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:27.367899 10402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:27.367985 10402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:27.368034 10402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:27.403681 10402 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:34143
I20260929 16:02:27.403774 10531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:34143 every 8 connection(s)
I20260929 16:02:27.405077 10402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:27.412161 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 10402
I20260929 16:02:27.412384 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:27.412977 24441 external_mini_cluster.cc:1428] Setting key aa09c9347c6768759fc54c4d5e1aac4e
Sep 29 16:02:27 dist-test-slave-cq88 krb5kdc[10295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:27.429495 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:27.407849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:50021 (local address 127.23.222.126:38367)
0929 16:02:27.408327 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:27.408334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:27.409221 (+ 887us) server_negotiation.cc:407] Connection header received
0929 16:02:27.411110 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:27.411116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:27.411214 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:27.411386 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:27.413646 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:27.414742 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:27.417482 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:27.417927 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:27.418046 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:27.422839 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:27.422845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:27.422849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:27.422890 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:27.425875 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:27.426746 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:27.426752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:27.426764 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:27.426897 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:27.427465 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:27.427471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:27.427477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:27.427772 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:27.427996 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:27.428807 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 16:02:27.429071 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":263,"threads_started":1}
I20260929 16:02:27.430583 10535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:27.408334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38367 (local address 127.23.222.65:50021)
0929 16:02:27.409053 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:27.409096 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:27.410827 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:27.411637 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:27.411646 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:27.412502 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:27.413384 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:27.413411 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:27.414991 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:27.414997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:27.417262 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:27.417284 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:27.418207 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:27.418217 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:27.418596 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:27.419517 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:27.419553 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:27.422530 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:27.426063 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:27.426075 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:27.426095 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:27.426600 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:27.427027 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:27.427032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:27.427037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:27.427338 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:27.427964 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:27.427973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:27.428377 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 16:02:27.428668 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 16:02:27.429297 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":273,"threads_started":1}
I20260929 16:02:27.432430 10532 heartbeater.cc:347] Connected to a master server at 127.23.222.126:38367
I20260929 16:02:27.432778 10532 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:27.433737 10532 heartbeater.cc:511] Master 127.23.222.126:38367 requested a full tablet report, sending...
I20260929 16:02:27.436517 10340 ts_manager.cc:194] Registered new tserver with Master: f4dabd2cc0db4ceb9a254844180dc686 (127.23.222.65:34143)
I20260929 16:02:27.438853 10340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:50021
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 16:02:27.474427 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38367
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:27.823884 10538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:27.824203 10538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:27.824340 10538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:27.831880 10538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:27.832007 10538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:27.832124 10538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:02:27.842459 10538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:27.843849 10538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:27.845132 10538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:27.859498 10543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:27.859521 10546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:27.859576 10544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:27.860519 10538 server_base.cc:1034] running on GCE node
I20260929 16:02:27.861186 10538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:27.862229 10538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:27.863443 10538 hybrid_clock.cc:648] HybridClock initialized: now 1790697747863386 us; error 41 us; skew 500 ppm
I20260929 16:02:27.863785 10538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:27 dist-test-slave-cq88 krb5kdc[10295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:27.867163 10538 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:02:27.868819 10538 webserver.cc:460] Webserver started at http://127.23.222.66:42467/ using document root <none> and password file <none>
I20260929 16:02:27.869570 10538 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:27.869727 10538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:27.870062 10538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:27.873409 10538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "0d9f669a349c4882917eacb3a194f84f"
format_stamp: "Formatted at 2026-09-29 16:02:27 on dist-test-slave-cq88"
server_key: "28e72050e41866e44aa5c3ca85482a13"
server_key_iv: "7a6d2a38245fe649f2323d9ccf1d70a4"
server_key_version: "encryptionkey@0"
I20260929 16:02:27.874264 10538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "0d9f669a349c4882917eacb3a194f84f"
format_stamp: "Formatted at 2026-09-29 16:02:27 on dist-test-slave-cq88"
server_key: "28e72050e41866e44aa5c3ca85482a13"
server_key_iv: "7a6d2a38245fe649f2323d9ccf1d70a4"
server_key_version: "encryptionkey@0"
I20260929 16:02:27.880185 10538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:02:27.883867 10553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:27.885294 10538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:27.885488 10538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "0d9f669a349c4882917eacb3a194f84f"
format_stamp: "Formatted at 2026-09-29 16:02:27 on dist-test-slave-cq88"
server_key: "28e72050e41866e44aa5c3ca85482a13"
server_key_iv: "7a6d2a38245fe649f2323d9ccf1d70a4"
server_key_version: "encryptionkey@0"
I20260929 16:02:27.885705 10538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:27.911767 10538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:27.916648 10538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:27.917008 10538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:27.918144 10538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:27.919834 10538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:27.919962 10538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:27.920125 10538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:27.920205 10538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:27.955775 10538 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:37735
I20260929 16:02:27.955919 10668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:37735 every 8 connection(s)
I20260929 16:02:27.957289 10538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:27.961099 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 10538
I20260929 16:02:27.961331 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:27.961989 24441 external_mini_cluster.cc:1428] Setting key 02cd0a7ace324cce608fe9e0af620039
Sep 29 16:02:27 dist-test-slave-cq88 krb5kdc[10295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697747, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:27.985617 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:27.960371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:60343 (local address 127.23.222.126:38367)
0929 16:02:27.960885 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:27.960892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:27.962299 (+ 1407us) server_negotiation.cc:407] Connection header received
0929 16:02:27.964992 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:27.964996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:27.965067 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:27.965168 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:27.967811 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:27.968668 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:27.971757 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:27.972256 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:27.972378 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:27.977093 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:27.977105 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:27.977114 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:27.977171 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:27.981374 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:27.982502 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:27.982512 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:27.982519 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:27.982710 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:27.983222 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:27.983231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:27.983277 (+ 46us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:27.983719 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:27.983897 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:27.984724 (+ 827us) server_negotiation.cc:299] Negotiation successful
0929 16:02:27.985065 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":243,"threads_started":1}
I20260929 16:02:27.986259 10672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:27.961027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38367 (local address 127.23.222.66:60343)
0929 16:02:27.962078 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:27.962139 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:27.964607 (+ 2468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:27.965517 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:27.965534 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:27.966496 (+ 962us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:27.967542 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:27.967576 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:27.968878 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:27.968887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:27.971482 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:27.971509 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:27.972595 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:27.972607 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:27.973034 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:27.973939 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:27.973980 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:27.976729 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:27.981640 (+ 4911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:27.981650 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:27.981672 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:27.982323 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:27.982848 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:27.982852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:27.982854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:27.983079 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:27.983877 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:27.983884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:27.984275 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 16:02:27.984538 (+ 263us) client_negotiation.cc:240] Negotiation successful
0929 16:02:27.985031 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":407,"threads_started":1}
I20260929 16:02:27.987958 10669 heartbeater.cc:347] Connected to a master server at 127.23.222.126:38367
I20260929 16:02:27.988291 10669 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:27.989216 10669 heartbeater.cc:511] Master 127.23.222.126:38367 requested a full tablet report, sending...
I20260929 16:02:27.990990 10340 ts_manager.cc:194] Registered new tserver with Master: 0d9f669a349c4882917eacb3a194f84f (127.23.222.66:37735)
I20260929 16:02:27.991948 10340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:60343
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 16:02:28.031958 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38367
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:40399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:28.390972 10675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:28.391351 10675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:28.391531 10675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:28.399544 10675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:28.399665 10675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:28.399834 10675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:02:28.412897 10675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:40399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:28.414206 10675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:28.415702 10675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:28.430562 10683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:28.430681 10680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:28.430581 10681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:28.431494 10675 server_base.cc:1034] running on GCE node
I20260929 16:02:28.432230 10675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:28.433329 10675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:28.434607 10675 hybrid_clock.cc:648] HybridClock initialized: now 1790697748434559 us; error 94 us; skew 500 ppm
I20260929 16:02:28.435076 10675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:28 dist-test-slave-cq88 krb5kdc[10295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697748, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:28.439019 10675 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:02:28.440975 10675 webserver.cc:460] Webserver started at http://127.23.222.67:32853/ using document root <none> and password file <none>
I20260929 16:02:28.441776 10675 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:28.441944 10675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:28.442359 10675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:28.442690 10532 heartbeater.cc:503] Master 127.23.222.126:38367 was elected leader, sending a full tablet report...
I20260929 16:02:28.446195 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "c9b0b55be696430f9604b1e0a83c9291"
format_stamp: "Formatted at 2026-09-29 16:02:28 on dist-test-slave-cq88"
server_key: "7351612a2b23eeaeed75f0bc983ef43f"
server_key_iv: "ad6e035b5ba6fff857c4ce04e4812139"
server_key_version: "encryptionkey@0"
I20260929 16:02:28.447052 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "c9b0b55be696430f9604b1e0a83c9291"
format_stamp: "Formatted at 2026-09-29 16:02:28 on dist-test-slave-cq88"
server_key: "7351612a2b23eeaeed75f0bc983ef43f"
server_key_iv: "ad6e035b5ba6fff857c4ce04e4812139"
server_key_version: "encryptionkey@0"
I20260929 16:02:28.453524 10675 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 16:02:28.457922 10690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:28.459936 10675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 16:02:28.460155 10675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "c9b0b55be696430f9604b1e0a83c9291"
format_stamp: "Formatted at 2026-09-29 16:02:28 on dist-test-slave-cq88"
server_key: "7351612a2b23eeaeed75f0bc983ef43f"
server_key_iv: "ad6e035b5ba6fff857c4ce04e4812139"
server_key_version: "encryptionkey@0"
I20260929 16:02:28.460403 10675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:28.474066 10675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:28.478430 10675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:28.478745 10675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:28.479831 10675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:28.481228 10675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:28.481333 10675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:28.481493 10675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:28.481567 10675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:28.516579 10675 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:43879
I20260929 16:02:28.516601 10804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:43879 every 8 connection(s)
I20260929 16:02:28.517944 10675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:28.521531 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 10675
I20260929 16:02:28.521765 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:28.522436 24441 external_mini_cluster.cc:1428] Setting key 597b4b000109c484c75fda96b214de15
Sep 29 16:02:28 dist-test-slave-cq88 krb5kdc[10295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697748, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:28.542586 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.520786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:51405 (local address 127.23.222.126:38367)
0929 16:02:28.521253 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.521259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.522501 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 16:02:28.525218 (+ 2717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.525223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.525333 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.525471 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:28.527198 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.528088 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.529816 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.530384 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.530492 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.535204 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:28.535212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:28.535218 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:28.535313 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:28.539211 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:28.540158 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:28.540165 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:28.540169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:28.540305 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:28.540748 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:28.540756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:28.540759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:28.541045 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:28.541215 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.541956 (+ 741us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.542186 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260929 16:02:28.543777 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.521352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38367 (local address 127.23.222.67:51405)
0929 16:02:28.522295 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:28.522356 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:28.524949 (+ 2593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:28.525778 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:28.525794 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:28.526366 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:28.526981 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.526999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.528254 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.528258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.529632 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.529650 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.530625 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.530635 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.531003 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.531864 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:28.531904 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:28.534889 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:28.539425 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:28.539436 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:28.539457 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:28.540028 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:28.540424 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:28.540429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:28.540433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:28.540634 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:28.541184 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:28.541192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:28.541578 (+ 386us) client_negotiation.cc:769] Sending connection context
0929 16:02:28.541856 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 16:02:28.542439 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":376,"threads_started":1}
I20260929 16:02:28.545400 10805 heartbeater.cc:347] Connected to a master server at 127.23.222.126:38367
I20260929 16:02:28.545724 10805 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:28.546465 10805 heartbeater.cc:511] Master 127.23.222.126:38367 requested a full tablet report, sending...
I20260929 16:02:28.548089 10340 ts_manager.cc:194] Registered new tserver with Master: c9b0b55be696430f9604b1e0a83c9291 (127.23.222.67:43879)
I20260929 16:02:28.548913 10340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:51405
I20260929 16:02:28.550735 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:28.568246 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.555015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.23.222.126:38367)
0929 16:02:28.555307 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.555313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.555560 (+ 247us) server_negotiation.cc:407] Connection header received
0929 16:02:28.555707 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.555711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.555786 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.555868 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:28.557300 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.558008 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.559423 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.559839 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.559950 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.562608 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:28.562613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:28.562616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:28.562651 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:28.565461 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:28.566260 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:28.566264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:28.566267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:28.566383 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:28.566755 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:28.566758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:28.566760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:28.567076 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:28.567361 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.567760 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.567905 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260929 16:02:28.572995 10339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56066:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:28.580981 10339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:28.628820 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.594639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:34143 (local address 127.0.0.1:42066)
0929 16:02:28.614423 (+ 19784us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:28.614488 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:28.614758 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:28.616980 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:28.616992 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:28.617039 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:28.617639 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.617657 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.621195 (+ 3538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.621204 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.624660 (+ 3456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.624693 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.627482 (+ 2789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.627493 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.627821 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.627902 (+ 81us) client_negotiation.cc:769] Sending connection context
0929 16:02:28.628085 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 16:02:28.628364 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19387,"threads_started":1}
I20260929 16:02:28.629750 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.594861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.23.222.65:34143)
0929 16:02:28.612443 (+ 17582us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.612457 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.615818 (+ 3361us) server_negotiation.cc:407] Connection header received
0929 16:02:28.615961 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.615971 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.616288 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.616554 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:28.617911 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.620143 (+ 2232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.624992 (+ 4849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.627173 (+ 2181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.627408 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.627757 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.629203 (+ 1446us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.629397 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17308,"thread_start_us":17108,"threads_started":1}
I20260929 16:02:28.634294 10466 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbc61de52f34730bc11d41abe5fbcb0 (DEFAULT_TABLE table=test-table [id=5e2a1405110148b8a2fd51e127488bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:28.636022 10466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbc61de52f34730bc11d41abe5fbcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:28.636229 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.595185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.23.222.67:43879)
0929 16:02:28.628349 (+ 33164us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.628360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.628401 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:02:28.628507 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.628516 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.628833 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.629729 (+ 896us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:28.630046 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.631522 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.633713 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.635072 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.635232 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.635517 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.635708 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.635880 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32960,"thread_start_us":32776,"threads_started":1}
I20260929 16:02:28.636266 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.595373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:43879 (local address 127.0.0.1:50396)
0929 16:02:28.620840 (+ 25467us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:28.620868 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:28.621035 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:28.629104 (+ 8069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:28.629109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:28.629144 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:28.629548 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.629567 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.631657 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.631663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.633525 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.633540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.635309 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.635317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.635501 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.635538 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:02:28.635661 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:02:28.635810 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25275,"mutex_wait_us":1,"thread_start_us":24994,"threads_started":1}
I20260929 16:02:28.637508 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.597419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:37735 (local address 127.0.0.1:53382)
0929 16:02:28.629149 (+ 31730us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:28.629169 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:28.629379 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:28.630316 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:28.630321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:28.630348 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:28.630665 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.630673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.632443 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.632448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.634525 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.634540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.636691 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.636699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.636870 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.636914 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 16:02:28.637023 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 16:02:28.637164 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31601,"thread_start_us":17615,"threads_started":1}
I20260929 16:02:28.637598 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.597722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.23.222.66:37735)
0929 16:02:28.628640 (+ 30918us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.628649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.629268 (+ 619us) server_negotiation.cc:407] Connection header received
0929 16:02:28.629590 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.629600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.629851 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.630060 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:28.630825 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.632287 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.634698 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.636211 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.636354 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.636545 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.637092 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.637278 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30765,"thread_start_us":30657,"threads_started":1}
I20260929 16:02:28.640535 10601 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbc61de52f34730bc11d41abe5fbcb0 (DEFAULT_TABLE table=test-table [id=5e2a1405110148b8a2fd51e127488bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:28.640547 10739 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbc61de52f34730bc11d41abe5fbcb0 (DEFAULT_TABLE table=test-table [id=5e2a1405110148b8a2fd51e127488bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:28.641855 10601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbc61de52f34730bc11d41abe5fbcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:28.642050 10739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbc61de52f34730bc11d41abe5fbcb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:28.654248 10825 tablet_bootstrap.cc:492] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: Bootstrap starting.
I20260929 16:02:28.656152 10824 tablet_bootstrap.cc:492] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: Bootstrap starting.
I20260929 16:02:28.658293 10826 tablet_bootstrap.cc:492] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: Bootstrap starting.
I20260929 16:02:28.658556 10825 tablet_bootstrap.cc:654] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:28.659430 10824 tablet_bootstrap.cc:654] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:28.659909 10825 log.cc:824] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:28.660660 10824 log.cc:824] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:28.661830 10826 tablet_bootstrap.cc:654] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:28.663035 10826 log.cc:824] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:28.663651 10824 tablet_bootstrap.cc:492] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: No bootstrap required, opened a new log
I20260929 16:02:28.663825 10825 tablet_bootstrap.cc:492] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: No bootstrap required, opened a new log
I20260929 16:02:28.664119 10825 ts_tablet_manager.cc:1397] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 16:02:28.664116 10824 ts_tablet_manager.cc:1397] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260929 16:02:28.666224 10826 tablet_bootstrap.cc:492] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: No bootstrap required, opened a new log
I20260929 16:02:28.666503 10826 ts_tablet_manager.cc:1397] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 16:02:28.672464 10824 raft_consensus.cc:348] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } }
I20260929 16:02:28.672902 10824 raft_consensus.cc:374] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:28.673009 10824 raft_consensus.cc:729] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4dabd2cc0db4ceb9a254844180dc686, State: Initialized, Role: FOLLOWER
I20260929 16:02:28.673723 10824 consensus_queue.cc:260] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } }
I20260929 16:02:28.675037 10825 raft_consensus.cc:348] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } }
I20260929 16:02:28.675513 10825 raft_consensus.cc:374] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:28.675598 10825 raft_consensus.cc:729] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d9f669a349c4882917eacb3a194f84f, State: Initialized, Role: FOLLOWER
I20260929 16:02:28.675840 10824 ts_tablet_manager.cc:1428] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260929 16:02:28.676196 10825 consensus_queue.cc:260] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } }
I20260929 16:02:28.677371 10669 heartbeater.cc:503] Master 127.23.222.126:38367 was elected leader, sending a full tablet report...
I20260929 16:02:28.678016 10825 ts_tablet_manager.cc:1428] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 16:02:28.678134 10826 raft_consensus.cc:348] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } }
I20260929 16:02:28.678508 10826 raft_consensus.cc:374] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:28.678644 10826 raft_consensus.cc:729] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b0b55be696430f9604b1e0a83c9291, State: Initialized, Role: FOLLOWER
I20260929 16:02:28.679315 10826 consensus_queue.cc:260] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } }
I20260929 16:02:28.680454 10805 heartbeater.cc:503] Master 127.23.222.126:38367 was elected leader, sending a full tablet report...
I20260929 16:02:28.682654 10826 ts_tablet_manager.cc:1428] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260929 16:02:28.709635 10670 tablet.cc:2369] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:28.719913 10831 raft_consensus.cc:482] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:28.720285 10831 raft_consensus.cc:504] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } }
I20260929 16:02:28.722203 10831 leader_election.cc:288] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4dabd2cc0db4ceb9a254844180dc686 (127.23.222.65:34143), c9b0b55be696430f9604b1e0a83c9291 (127.23.222.67:43879)
I20260929 16:02:28.729710 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.722801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:35965 (local address 127.23.222.67:43879)
0929 16:02:28.723157 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.723165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.723606 (+ 441us) server_negotiation.cc:407] Connection header received
0929 16:02:28.723855 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.723860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.723923 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.724029 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:28.724598 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.725364 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.727594 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.728650 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.728831 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.728911 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.729304 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.729437 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260929 16:02:28.729935 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.722509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:43879 (local address 127.23.222.66:35965)
0929 16:02:28.723472 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:28.723492 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:28.723693 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:28.724047 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:28.724052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:28.724077 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:28.724443 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.724452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.725508 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.725516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.727384 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.727404 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.728894 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.728901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.729098 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.729141 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 16:02:28.729264 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 16:02:28.729420 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":102,"thread_start_us":565,"threads_started":1}
I20260929 16:02:28.730094 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.722694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:34143 (local address 127.23.222.66:44159)
0929 16:02:28.723776 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:28.723791 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:28.723950 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:28.724304 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:28.724308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:28.724337 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:28.724734 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.724742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.725782 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.725787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.727478 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:28.727514 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.729137 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.729142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:28.729304 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.729327 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 16:02:28.729437 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 16:02:28.729613 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":3,"thread_start_us":682,"threads_started":1}
I20260929 16:02:28.730279 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.722764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:44159 (local address 127.23.222.65:34143)
0929 16:02:28.723009 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.723016 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.723885 (+ 869us) server_negotiation.cc:407] Connection header received
0929 16:02:28.724100 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.724104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.724165 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.724275 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:28.724888 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.725624 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.727755 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.728876 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.729029 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.729102 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.729478 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.729613 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:02:28.731104 10759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbc61de52f34730bc11d41abe5fbcb0" candidate_uuid: "0d9f669a349c4882917eacb3a194f84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9b0b55be696430f9604b1e0a83c9291" is_pre_election: true
I20260929 16:02:28.731202 10486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbc61de52f34730bc11d41abe5fbcb0" candidate_uuid: "0d9f669a349c4882917eacb3a194f84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4dabd2cc0db4ceb9a254844180dc686" is_pre_election: true
I20260929 16:02:28.731760 10759 raft_consensus.cc:2454] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9f669a349c4882917eacb3a194f84f in term 0.
I20260929 16:02:28.731856 10486 raft_consensus.cc:2454] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d9f669a349c4882917eacb3a194f84f in term 0.
I20260929 16:02:28.732542 10555 leader_election.cc:302] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9f669a349c4882917eacb3a194f84f, f4dabd2cc0db4ceb9a254844180dc686; no voters:
I20260929 16:02:28.732981 10831 raft_consensus.cc:2781] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:28.733139 10831 raft_consensus.cc:482] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:28.733246 10831 raft_consensus.cc:3037] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:28.735545 10831 raft_consensus.cc:504] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } }
I20260929 16:02:28.736228 10831 leader_election.cc:288] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [CANDIDATE]: Term 1 election: Requested vote from peers f4dabd2cc0db4ceb9a254844180dc686 (127.23.222.65:34143), c9b0b55be696430f9604b1e0a83c9291 (127.23.222.67:43879)
I20260929 16:02:28.736779 10486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbc61de52f34730bc11d41abe5fbcb0" candidate_uuid: "0d9f669a349c4882917eacb3a194f84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4dabd2cc0db4ceb9a254844180dc686"
I20260929 16:02:28.736825 10759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6dbc61de52f34730bc11d41abe5fbcb0" candidate_uuid: "0d9f669a349c4882917eacb3a194f84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9b0b55be696430f9604b1e0a83c9291"
I20260929 16:02:28.737015 10486 raft_consensus.cc:3037] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:28.737069 10759 raft_consensus.cc:3037] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:28.739447 10486 raft_consensus.cc:2454] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9f669a349c4882917eacb3a194f84f in term 1.
I20260929 16:02:28.739472 10759 raft_consensus.cc:2454] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d9f669a349c4882917eacb3a194f84f in term 1.
I20260929 16:02:28.739969 10555 leader_election.cc:302] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d9f669a349c4882917eacb3a194f84f, f4dabd2cc0db4ceb9a254844180dc686; no voters:
I20260929 16:02:28.740322 10831 raft_consensus.cc:2781] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:28.740824 10831 raft_consensus.cc:686] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 1 LEADER]: Becoming Leader. State: Replica: 0d9f669a349c4882917eacb3a194f84f, State: Running, Role: LEADER
I20260929 16:02:28.741353 10831 consensus_queue.cc:237] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } }
I20260929 16:02:28.746563 10339 catalog_manager.cc:5495] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d9f669a349c4882917eacb3a194f84f (127.23.222.66). New cstate: current_term: 1 leader_uuid: "0d9f669a349c4882917eacb3a194f84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d9f669a349c4882917eacb3a194f84f" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 37735 } health_report { overall_health: HEALTHY } } }
W20260929 16:02:28.769562 10806 tablet.cc:2369] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:28.772280 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.766680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.23.222.66:37735)
0929 16:02:28.766868 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.766872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.766889 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:02:28.766983 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.766986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.767071 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.767208 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:28.767685 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.768395 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.769868 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.770518 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.770620 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.771105 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:28.771686 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:28.771822 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.772026 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.772067 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:02:28.781016 10486 raft_consensus.cc:1261] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [term 1 FOLLOWER]: Refusing update from remote peer 0d9f669a349c4882917eacb3a194f84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:28.781019 10759 raft_consensus.cc:1261] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [term 1 FOLLOWER]: Refusing update from remote peer 0d9f669a349c4882917eacb3a194f84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:28.782111 10835 consensus_queue.cc:1035] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4dabd2cc0db4ceb9a254844180dc686" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:28.782500 10831 consensus_queue.cc:1035] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9b0b55be696430f9604b1e0a83c9291" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:28.813078 10340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:28.813418 10340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56066:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:28.817894 10340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f284506ede6a46e8b4905f421d114d2e: Sending DeleteTablet for 3 replicas of tablet 6dbc61de52f34730bc11d41abe5fbcb0
I20260929 16:02:28.819180 10739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dbc61de52f34730bc11d41abe5fbcb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:28 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:50396
I20260929 16:02:28.819239 10466 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dbc61de52f34730bc11d41abe5fbcb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:28 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:42066
I20260929 16:02:28.819923 10601 tablet_service.cc:1514] Processing DeleteTablet for tablet 6dbc61de52f34730bc11d41abe5fbcb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:28 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:53382
I20260929 16:02:28.823148 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 10402
I20260929 16:02:28.834812 10849 tablet_replica.cc:330] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: stopping tablet replica
I20260929 16:02:28.835434 10849 raft_consensus.cc:2229] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:28.836071 10849 raft_consensus.cc:2258] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:28.838732 10849 ts_tablet_manager.cc:1905] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:28.840390 10848 tablet_replica.cc:330] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: stopping tablet replica
I20260929 16:02:28.841086 10850 tablet_replica.cc:330] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: stopping tablet replica
I20260929 16:02:28.841621 10850 raft_consensus.cc:2229] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:28.842199 10848 raft_consensus.cc:2229] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:28.843122 10850 raft_consensus.cc:2258] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:28.843750 10848 raft_consensus.cc:2258] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:28.846307 10848 ts_tablet_manager.cc:1905] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:28.847308 10849 ts_tablet_manager.cc:1918] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:28.847502 10849 log.cc:1196] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/6dbc61de52f34730bc11d41abe5fbcb0
Sep 29 16:02:28 dist-test-slave-cq88 krb5kdc[10295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 16:02:28.848088 10849 ts_tablet_manager.cc:1939] T 6dbc61de52f34730bc11d41abe5fbcb0 P f4dabd2cc0db4ceb9a254844180dc686: Deleting consensus metadata
I20260929 16:02:28.849283 10850 ts_tablet_manager.cc:1905] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:28.850193 10325 catalog_manager.cc:4841] TS f4dabd2cc0db4ceb9a254844180dc686 (127.23.222.65:34143): tablet 6dbc61de52f34730bc11d41abe5fbcb0 (table test-table [id=5e2a1405110148b8a2fd51e127488bec]) successfully deleted
I20260929 16:02:28.856830 10848 ts_tablet_manager.cc:1918] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:28.857043 10848 log.cc:1196] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/6dbc61de52f34730bc11d41abe5fbcb0
I20260929 16:02:28.857511 10850 ts_tablet_manager.cc:1918] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:28.857605 10848 ts_tablet_manager.cc:1939] T 6dbc61de52f34730bc11d41abe5fbcb0 P c9b0b55be696430f9604b1e0a83c9291: Deleting consensus metadata
I20260929 16:02:28.857712 10850 log.cc:1196] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/6dbc61de52f34730bc11d41abe5fbcb0
I20260929 16:02:28.858301 10850 ts_tablet_manager.cc:1939] T 6dbc61de52f34730bc11d41abe5fbcb0 P 0d9f669a349c4882917eacb3a194f84f: Deleting consensus metadata
I20260929 16:02:28.860127 10324 catalog_manager.cc:4841] TS 0d9f669a349c4882917eacb3a194f84f (127.23.222.66:37735): tablet 6dbc61de52f34730bc11d41abe5fbcb0 (table test-table [id=5e2a1405110148b8a2fd51e127488bec]) successfully deleted
I20260929 16:02:28.860160 10326 catalog_manager.cc:4841] TS c9b0b55be696430f9604b1e0a83c9291 (127.23.222.67:43879): tablet 6dbc61de52f34730bc11d41abe5fbcb0 (table test-table [id=5e2a1405110148b8a2fd51e127488bec]) successfully deleted
I20260929 16:02:28.860745 10819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.824484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42074 (local address 127.23.222.65:34143)
0929 16:02:28.824721 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.824730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.829157 (+ 4427us) server_negotiation.cc:407] Connection header received
0929 16:02:28.830175 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.830185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.830303 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.830411 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:28.835257 (+ 4846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.836950 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.841763 (+ 4813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.842463 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.842579 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.851844 (+ 9265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:28.851863 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:28.851886 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:28.851946 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:28.856016 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:28.857285 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:28.857296 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:28.857300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:28.857483 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:28.858221 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:28.858226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:28.858229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:28.858866 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:28.859186 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.859994 (+ 808us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.860257 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 16:02:28.861919 10526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:28.960479 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 10402
I20260929 16:02:28.980034 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 10538
Sep 29 16:02:28 dist-test-slave-cq88 krb5kdc[10295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 16:02:29.000247 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:28.980884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.23.222.66:37735)
0929 16:02:28.981079 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:28.981087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:28.981111 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:02:28.981332 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:28.981339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:28.981425 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:28.981534 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:28.983182 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.984535 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.986559 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:28.987558 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:28.987674 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:28.993552 (+ 5878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:28.993565 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:28.993579 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:28.993619 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:28.997095 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:28.998098 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:28.998105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:28.998108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:28.998257 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:28.998699 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:28.998702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:28.998704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:28.999083 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:28.999412 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:28.999719 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 16:02:28.999915 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:02:29.001081 10664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:29.120090 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 10538
I20260929 16:02:29.140013 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 10675
Sep 29 16:02:29 dist-test-slave-cq88 krb5kdc[10295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 16:02:29.156812 10820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:29.140953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.23.222.67:43879)
0929 16:02:29.141171 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:29.141177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:29.141196 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:02:29.141266 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:29.141269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:29.141332 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:29.141460 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:29.142809 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:29.143697 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:29.145151 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:29.145614 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:29.145771 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:29.150563 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:29.150580 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:29.150597 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:29.150638 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:29.153733 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:29.154533 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:29.154541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:29.154543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:29.154683 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:29.155151 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:29.155157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:29.155160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:29.155619 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:29.155943 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:29.156237 (+ 294us) server_negotiation.cc:299] Negotiation successful
0929 16:02:29.156465 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:02:29.157554 10800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:29.261957 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 10675
I20260929 16:02:29.279476 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 10308
I20260929 16:02:29.280597 10371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:29.396358 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 10308
2026-09-29T16:02:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4534 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T16:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:29Z Disabled control of system clock
I20260929 16:02:29.478751 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:46759
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:46759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:29.830147 10859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:29.830482 10859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:29.830569 10859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:29.838194 10859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:29.838322 10859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:29.838359 10859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:29.838413 10859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:29.838465 10859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:29.848109 10859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:46759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:46759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:29.849411 10859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:29.851190 10859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:29.864500 10865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:29.864599 10864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:29.864728 10867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:29.865618 10859 server_base.cc:1034] running on GCE node
I20260929 16:02:29.866500 10859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:29.868095 10859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:29.869321 10859 hybrid_clock.cc:648] HybridClock initialized: now 1790697749869258 us; error 64 us; skew 500 ppm
I20260929 16:02:29.869679 10859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:29.872068 10859 webserver.cc:460] Webserver started at http://127.23.222.126:45379/ using document root <none> and password file <none>
I20260929 16:02:29.872776 10859 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:29.872908 10859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:29.873255 10859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:29.876360 10859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "a245234bdcff489188ddfe75516c08c1"
format_stamp: "Formatted at 2026-09-29 16:02:29 on dist-test-slave-cq88"
server_key: "ae1ef7b4ce0323c515d4d40f4109770f"
server_key_iv: "c936911a91d18fa840b2db63f347bf0e"
server_key_version: "encryptionkey@0"
I20260929 16:02:29.877202 10859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "a245234bdcff489188ddfe75516c08c1"
format_stamp: "Formatted at 2026-09-29 16:02:29 on dist-test-slave-cq88"
server_key: "ae1ef7b4ce0323c515d4d40f4109770f"
server_key_iv: "c936911a91d18fa840b2db63f347bf0e"
server_key_version: "encryptionkey@0"
I20260929 16:02:29.882622 10859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 16:02:29.886473 10873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:29.888080 10859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:29.888294 10859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "a245234bdcff489188ddfe75516c08c1"
format_stamp: "Formatted at 2026-09-29 16:02:29 on dist-test-slave-cq88"
server_key: "ae1ef7b4ce0323c515d4d40f4109770f"
server_key_iv: "c936911a91d18fa840b2db63f347bf0e"
server_key_version: "encryptionkey@0"
I20260929 16:02:29.888540 10859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:29.917287 10859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:29.918205 10859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:29.918514 10859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:29.939034 10859 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:46759
I20260929 16:02:29.939078 10924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:46759 every 8 connection(s)
I20260929 16:02:29.940486 10859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:29.944526 10925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:29.944729 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 10859
I20260929 16:02:29.944906 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:29.945363 24441 external_mini_cluster.cc:1428] Setting key 8434dd9ee42909ef3ffefe256b235d25
I20260929 16:02:29.957719 10925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1: Bootstrap starting.
I20260929 16:02:29.958847 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:29.947229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.23.222.126:46759)
0929 16:02:29.948329 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:29.948340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:29.948381 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:02:29.949970 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:29.949993 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:29.950473 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:29.950898 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:29.951534 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:29.952804 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:29.954289 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:29.955031 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:29.955276 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:29.956517 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:29.956525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:29.956538 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:29.957090 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:29.957171 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:29.957192 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:29.957315 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:29.957554 (+ 239us) server_negotiation.cc:299] Negotiation successful
0929 16:02:29.957898 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":397,"threads_started":1}
I20260929 16:02:29.961089 10925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:29.962247 10925 log.cc:824] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:29.965103 10925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1: No bootstrap required, opened a new log
I20260929 16:02:29.972934 10925 raft_consensus.cc:348] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a245234bdcff489188ddfe75516c08c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46759 } }
I20260929 16:02:29.973356 10925 raft_consensus.cc:374] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:29.973452 10925 raft_consensus.cc:729] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a245234bdcff489188ddfe75516c08c1, State: Initialized, Role: FOLLOWER
I20260929 16:02:29.974045 10925 consensus_queue.cc:260] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a245234bdcff489188ddfe75516c08c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46759 } }
I20260929 16:02:29.974304 10925 raft_consensus.cc:388] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:29.974449 10925 raft_consensus.cc:482] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:29.974640 10925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:29.976975 10925 raft_consensus.cc:504] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a245234bdcff489188ddfe75516c08c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46759 } }
I20260929 16:02:29.977509 10925 leader_election.cc:302] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a245234bdcff489188ddfe75516c08c1; no voters:
I20260929 16:02:29.978250 10925 leader_election.cc:288] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:29.978550 10930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:29.979664 10930 raft_consensus.cc:686] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [term 1 LEADER]: Becoming Leader. State: Replica: a245234bdcff489188ddfe75516c08c1, State: Running, Role: LEADER
I20260929 16:02:29.980211 10930 consensus_queue.cc:237] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a245234bdcff489188ddfe75516c08c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46759 } }
I20260929 16:02:29.980669 10925 sys_catalog.cc:564] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:29.983353 10931 sys_catalog.cc:455] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a245234bdcff489188ddfe75516c08c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a245234bdcff489188ddfe75516c08c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46759 } } }
I20260929 16:02:29.983392 10932 sys_catalog.cc:455] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a245234bdcff489188ddfe75516c08c1. Latest consensus state: current_term: 1 leader_uuid: "a245234bdcff489188ddfe75516c08c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a245234bdcff489188ddfe75516c08c1" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 46759 } } }
I20260929 16:02:29.983757 10931 sys_catalog.cc:458] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:29.983843 10932 sys_catalog.cc:458] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:29.986763 10936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:29.991688 10936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:30.000665 10936 catalog_manager.cc:1347] Generated new cluster ID: 00c0be041aba4978ac144fc356962b29
I20260929 16:02:30.000833 10936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:30.040241 10936 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:30.042307 10936 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:30.052455 10936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1: Generated new TSK 0
I20260929 16:02:30.053429 10936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:30.058580 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46759
--builtin_ntp_servers=127.23.222.84:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:30.464311 10950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:30.464639 10950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:30.464809 10950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:30.472401 10950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:30.472522 10950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:30.472705 10950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:02:30.482633 10950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:30.484052 10950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:30.485886 10950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:30.499687 10955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:30.499725 10956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:30.500674 10958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:30.500999 10950 server_base.cc:1034] running on GCE node
I20260929 16:02:30.501636 10950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:30.502579 10950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:30.503785 10950 hybrid_clock.cc:648] HybridClock initialized: now 1790697750503743 us; error 39 us; skew 500 ppm
I20260929 16:02:30.504135 10950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:30.506456 10950 webserver.cc:460] Webserver started at http://127.23.222.65:34777/ using document root <none> and password file <none>
I20260929 16:02:30.507195 10950 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:30.507370 10950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:30.507714 10950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:30.510740 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "dee423a846f54282a830bcd9c09a1f7d"
format_stamp: "Formatted at 2026-09-29 16:02:30 on dist-test-slave-cq88"
server_key: "c455fe854dbc02729d9e91f8c7bc4ae7"
server_key_iv: "9f12daa43d95ebc0d265a976e530cb87"
server_key_version: "encryptionkey@0"
I20260929 16:02:30.511664 10950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "dee423a846f54282a830bcd9c09a1f7d"
format_stamp: "Formatted at 2026-09-29 16:02:30 on dist-test-slave-cq88"
server_key: "c455fe854dbc02729d9e91f8c7bc4ae7"
server_key_iv: "9f12daa43d95ebc0d265a976e530cb87"
server_key_version: "encryptionkey@0"
I20260929 16:02:30.517212 10950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:02:30.520928 10964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:30.522452 10950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:02:30.522675 10950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "dee423a846f54282a830bcd9c09a1f7d"
format_stamp: "Formatted at 2026-09-29 16:02:30 on dist-test-slave-cq88"
server_key: "c455fe854dbc02729d9e91f8c7bc4ae7"
server_key_iv: "9f12daa43d95ebc0d265a976e530cb87"
server_key_version: "encryptionkey@0"
I20260929 16:02:30.522946 10950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:30.535920 10950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:30.536832 10950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:30.537164 10950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:30.538329 10950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:30.540114 10950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:30.540241 10950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:30.540385 10950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:30.540486 10950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:30.576226 10950 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:45633
I20260929 16:02:30.576340 11077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:45633 every 8 connection(s)
I20260929 16:02:30.577634 10950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:30.580416 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 10950
I20260929 16:02:30.580614 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:30.581198 24441 external_mini_cluster.cc:1428] Setting key ee7fd4af67962858b7b4bbd2ed9660cd
I20260929 16:02:30.584481 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46759
--builtin_ntp_servers=127.23.222.84:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:30.594527 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:30.580630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:44305 (local address 127.23.222.126:46759)
0929 16:02:30.581207 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:30.581214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:30.582853 (+ 1639us) server_negotiation.cc:407] Connection header received
0929 16:02:30.585668 (+ 2815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:30.585672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:30.585745 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:30.585838 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:30.587444 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:30.588282 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:30.590960 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:30.591492 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:30.591616 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:30.593467 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:30.593475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:30.593482 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:30.593513 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:30.593570 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:30.593589 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:30.593688 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:30.593951 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 16:02:30.594080 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":303,"threads_started":1}
I20260929 16:02:30.595513 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:30.581372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:46759 (local address 127.23.222.65:44305)
0929 16:02:30.582689 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:30.582732 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:30.585306 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:30.586140 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:30.586156 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:30.586327 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:30.587212 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:30.587271 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:30.588457 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:30.588466 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:30.590764 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:30.590794 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:30.591777 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:30.591787 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:30.592209 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:30.593029 (+ 820us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:30.593036 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:30.593109 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:30.593233 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:30.593694 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:30.593703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:30.593709 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:02:30.593890 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 16:02:30.594221 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":574,"threads_started":1}
I20260929 16:02:30.597352 11078 heartbeater.cc:347] Connected to a master server at 127.23.222.126:46759
I20260929 16:02:30.597710 11078 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:30.598507 11078 heartbeater.cc:511] Master 127.23.222.126:46759 requested a full tablet report, sending...
I20260929 16:02:30.600819 10890 ts_manager.cc:194] Registered new tserver with Master: dee423a846f54282a830bcd9c09a1f7d (127.23.222.65:45633)
I20260929 16:02:30.602433 10890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.65:44305
W20260929 16:02:30.937103 11082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:30.937414 11082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:30.937510 11082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:30.945286 11082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:30.945403 11082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:30.945595 11082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:02:30.955382 11082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:30.956770 11082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:30.959323 11082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:30.974020 11087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:30.974040 11088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:30.974066 11090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:30.975010 11082 server_base.cc:1034] running on GCE node
I20260929 16:02:30.975687 11082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:30.976647 11082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:30.977872 11082 hybrid_clock.cc:648] HybridClock initialized: now 1790697750977816 us; error 45 us; skew 500 ppm
I20260929 16:02:30.978199 11082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:30.980700 11082 webserver.cc:460] Webserver started at http://127.23.222.66:40223/ using document root <none> and password file <none>
I20260929 16:02:30.981401 11082 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:30.981529 11082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:30.981886 11082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:30.984993 11082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "cd503c29398f4ba8b8775b2ab1a188f8"
format_stamp: "Formatted at 2026-09-29 16:02:30 on dist-test-slave-cq88"
server_key: "d475d17f0d49ed40f8159b59a26be989"
server_key_iv: "7f87f1c5cda6dd930d713431741ecfb6"
server_key_version: "encryptionkey@0"
I20260929 16:02:30.985759 11082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "cd503c29398f4ba8b8775b2ab1a188f8"
format_stamp: "Formatted at 2026-09-29 16:02:30 on dist-test-slave-cq88"
server_key: "d475d17f0d49ed40f8159b59a26be989"
server_key_iv: "7f87f1c5cda6dd930d713431741ecfb6"
server_key_version: "encryptionkey@0"
I20260929 16:02:30.991159 11082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:02:30.994971 11096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:30.996443 11082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:02:30.996628 11082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "cd503c29398f4ba8b8775b2ab1a188f8"
format_stamp: "Formatted at 2026-09-29 16:02:30 on dist-test-slave-cq88"
server_key: "d475d17f0d49ed40f8159b59a26be989"
server_key_iv: "7f87f1c5cda6dd930d713431741ecfb6"
server_key_version: "encryptionkey@0"
I20260929 16:02:30.996846 11082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:31.018021 11082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:31.019047 11082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:31.019553 11082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:31.020768 11082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:31.022433 11082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:31.022548 11082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:31.022639 11082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:31.022713 11082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:31.059718 11082 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:35315
I20260929 16:02:31.059810 11208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:35315 every 8 connection(s)
I20260929 16:02:31.061147 11082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:31.064033 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 11082
I20260929 16:02:31.064216 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:31.064663 24441 external_mini_cluster.cc:1428] Setting key fe5ffb552763c76ad23fb1738841c3a3
I20260929 16:02:31.067917 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46759
--builtin_ntp_servers=127.23.222.84:35035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:31.075116 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.063830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:55399 (local address 127.23.222.126:46759)
0929 16:02:31.064047 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.064054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:31.065773 (+ 1719us) server_negotiation.cc:407] Connection header received
0929 16:02:31.068061 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:31.068069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:31.068174 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:31.068261 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:31.069734 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.070460 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.072174 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.072596 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.072742 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.074256 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:31.074261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:31.074265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:31.074285 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:31.074316 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:31.074328 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:31.074435 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:31.074660 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 16:02:31.074768 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 16:02:31.076356 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.064517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:46759 (local address 127.23.222.66:55399)
0929 16:02:31.065594 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.065649 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:31.067737 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:31.068530 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:31.068545 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:31.068675 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:31.069554 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.069582 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.070635 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.070641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.072016 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.072029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.072841 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.072848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.073106 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.073890 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:31.073893 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:31.073942 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:31.074033 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:31.074425 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:31.074437 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:31.074447 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 16:02:31.074646 (+ 199us) client_negotiation.cc:240] Negotiation successful
0929 16:02:31.075105 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":333,"threads_started":1}
I20260929 16:02:31.078214 11209 heartbeater.cc:347] Connected to a master server at 127.23.222.126:46759
I20260929 16:02:31.078542 11209 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:31.079411 11209 heartbeater.cc:511] Master 127.23.222.126:46759 requested a full tablet report, sending...
I20260929 16:02:31.080996 10890 ts_manager.cc:194] Registered new tserver with Master: cd503c29398f4ba8b8775b2ab1a188f8 (127.23.222.66:35315)
I20260929 16:02:31.081964 10890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.66:55399
W20260929 16:02:31.425843 11212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:31.426203 11212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:31.426373 11212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:31.434623 11212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:31.434803 11212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:31.435015 11212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:02:31.445667 11212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:46759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:31.447077 11212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:31.448997 11212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:31.463425 11218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:31.463537 11221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:31.463452 11219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:31.464545 11212 server_base.cc:1034] running on GCE node
I20260929 16:02:31.465229 11212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:31.466499 11212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:31.467794 11212 hybrid_clock.cc:648] HybridClock initialized: now 1790697751467716 us; error 67 us; skew 500 ppm
I20260929 16:02:31.468201 11212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:31.470765 11212 webserver.cc:460] Webserver started at http://127.23.222.67:35959/ using document root <none> and password file <none>
I20260929 16:02:31.471637 11212 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:31.471773 11212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:31.472108 11212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:31.475203 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "7b13a2b746c94d2ca553b7b582a49f4d"
format_stamp: "Formatted at 2026-09-29 16:02:31 on dist-test-slave-cq88"
server_key: "723ae95f8ed2bf18360f03dfa8787555"
server_key_iv: "f2d75de7c98ff329b8cc4f504eda202e"
server_key_version: "encryptionkey@0"
I20260929 16:02:31.476161 11212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "7b13a2b746c94d2ca553b7b582a49f4d"
format_stamp: "Formatted at 2026-09-29 16:02:31 on dist-test-slave-cq88"
server_key: "723ae95f8ed2bf18360f03dfa8787555"
server_key_iv: "f2d75de7c98ff329b8cc4f504eda202e"
server_key_version: "encryptionkey@0"
I20260929 16:02:31.482024 11212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:02:31.485803 11227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:31.487349 11212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:02:31.487556 11212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "7b13a2b746c94d2ca553b7b582a49f4d"
format_stamp: "Formatted at 2026-09-29 16:02:31 on dist-test-slave-cq88"
server_key: "723ae95f8ed2bf18360f03dfa8787555"
server_key_iv: "f2d75de7c98ff329b8cc4f504eda202e"
server_key_version: "encryptionkey@0"
I20260929 16:02:31.487795 11212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:31.519661 11212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:31.520531 11212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:31.520844 11212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:31.521880 11212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:31.523473 11212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:31.523594 11212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:31.523725 11212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:31.523813 11212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:31.558420 11212 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:43385
I20260929 16:02:31.558509 11339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:43385 every 8 connection(s)
I20260929 16:02:31.559790 11212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:31.567710 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 11212
I20260929 16:02:31.567906 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:31.568377 24441 external_mini_cluster.cc:1428] Setting key 5810c375a4f895321c2529f582525f7f
I20260929 16:02:31.573722 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.562489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:42117 (local address 127.23.222.126:46759)
0929 16:02:31.562702 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.562707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:31.564133 (+ 1426us) server_negotiation.cc:407] Connection header received
0929 16:02:31.566072 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:31.566079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:31.566178 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:31.566303 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:31.567549 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.568551 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.570582 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.571035 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.571142 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.572822 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:31.572827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:31.572835 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:31.572853 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:31.572888 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:31.572900 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:31.572996 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:31.573317 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 16:02:31.573425 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 16:02:31.574731 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.562905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:46759 (local address 127.23.222.67:42117)
0929 16:02:31.563948 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.563993 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:31.565765 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:31.566547 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:31.566555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:31.566643 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:31.567326 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.567355 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.568723 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.568730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.570428 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.570449 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.571383 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.571395 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.571671 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.572470 (+ 799us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:31.572473 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:31.572516 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:31.572603 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:31.573021 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:31.573032 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:31.573040 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 16:02:31.573225 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 16:02:31.573574 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":390,"threads_started":1}
I20260929 16:02:31.576354 11340 heartbeater.cc:347] Connected to a master server at 127.23.222.126:46759
I20260929 16:02:31.576679 11340 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:31.577461 11340 heartbeater.cc:511] Master 127.23.222.126:46759 requested a full tablet report, sending...
I20260929 16:02:31.579129 10890 ts_manager.cc:194] Registered new tserver with Master: 7b13a2b746c94d2ca553b7b582a49f4d (127.23.222.67:43385)
I20260929 16:02:31.580284 10890 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.67:42117
I20260929 16:02:31.584286 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:31.595418 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.588875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.23.222.126:46759)
0929 16:02:31.589186 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.589191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:31.589639 (+ 448us) server_negotiation.cc:407] Connection header received
0929 16:02:31.589757 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:31.589761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:31.589832 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:31.589924 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:31.590432 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.591154 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.593000 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.593548 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.593650 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.594690 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:31.594695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:31.594699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:31.594716 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:31.594761 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:31.594773 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:31.594880 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:31.595069 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 16:02:31.595116 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260929 16:02:31.599527 10890 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 16:02:31.606048 11078 heartbeater.cc:503] Master 127.23.222.126:46759 was elected leader, sending a full tablet report...
W20260929 16:02:31.607195 10890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:31.645414 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.620304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35776 (local address 127.23.222.67:43385)
0929 16:02:31.636015 (+ 15711us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.636026 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:31.636066 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:02:31.636182 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:31.636198 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:31.636490 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:31.637497 (+ 1007us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:31.637806 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.639521 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.642695 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.644160 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.644318 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.644632 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:31.644901 (+ 269us) server_negotiation.cc:299] Negotiation successful
0929 16:02:31.645073 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15516,"thread_start_us":15358,"threads_started":1}
I20260929 16:02:31.645438 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.620221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:43385 (local address 127.0.0.1:35776)
0929 16:02:31.634635 (+ 14414us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.634702 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:31.635170 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:31.636729 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:31.636737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:31.636775 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:31.637266 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.637282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.639690 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.639695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.642434 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.642450 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.644380 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.644386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.644620 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.644677 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 16:02:31.644835 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 16:02:31.645033 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14024,"mutex_wait_us":35,"thread_start_us":15828,"threads_started":1}
I20260929 16:02:31.647140 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.621070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.23.222.65:45633)
0929 16:02:31.633493 (+ 12423us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.633508 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:31.636445 (+ 2937us) server_negotiation.cc:407] Connection header received
0929 16:02:31.636758 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:31.636766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:31.636991 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:31.637179 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:31.638126 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.639698 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.643752 (+ 4054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.645275 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.645440 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.645777 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:31.646558 (+ 781us) server_negotiation.cc:299] Negotiation successful
0929 16:02:31.646753 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12131,"thread_start_us":12456,"threads_started":1}
I20260929 16:02:31.647974 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.620640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45633 (local address 127.0.0.1:38440)
0929 16:02:31.636310 (+ 15670us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.636330 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:31.636554 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:31.637481 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:31.637485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:31.637511 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:31.637907 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.637920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.639922 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.639927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.643530 (+ 3603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.643543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.646121 (+ 2578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.646129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.646334 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.646379 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 16:02:31.647463 (+ 1084us) client_negotiation.cc:240] Negotiation successful
0929 16:02:31.647620 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15549,"mutex_wait_us":1422,"thread_start_us":13394,"threads_started":1}
I20260929 16:02:31.650272 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.636688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35315 (local address 127.0.0.1:41594)
0929 16:02:31.641477 (+ 4789us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.641497 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:31.641614 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:31.643024 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:31.643030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:31.643056 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:31.643591 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.643605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.645514 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.645520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.647611 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.647625 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.649457 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.649463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.649651 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.649684 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:02:31.649782 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 16:02:31.649950 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4611,"thread_start_us":4461,"threads_started":1}
I20260929 16:02:31.650399 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.636715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.23.222.66:35315)
0929 16:02:31.642116 (+ 5401us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.642126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:31.642154 (+ 28us) server_negotiation.cc:407] Connection header received
0929 16:02:31.642250 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:31.642258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:31.642550 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:31.642780 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:31.643814 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.645349 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.647829 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.649243 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.649399 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.649632 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:31.649870 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 16:02:31.650041 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5237,"thread_start_us":5124,"threads_started":1}
I20260929 16:02:31.650233 11275 tablet_service.cc:1467] Processing CreateTablet for tablet 342c1044c2914cb6b95d9d37c6ec9f23 (DEFAULT_TABLE table=test-table [id=5bc3f5c2f82046079b490991f633f8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:31.651993 11275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342c1044c2914cb6b95d9d37c6ec9f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:31.652069 11012 tablet_service.cc:1467] Processing CreateTablet for tablet 342c1044c2914cb6b95d9d37c6ec9f23 (DEFAULT_TABLE table=test-table [id=5bc3f5c2f82046079b490991f633f8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:31.653599 11012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342c1044c2914cb6b95d9d37c6ec9f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:31.654488 11144 tablet_service.cc:1467] Processing CreateTablet for tablet 342c1044c2914cb6b95d9d37c6ec9f23 (DEFAULT_TABLE table=test-table [id=5bc3f5c2f82046079b490991f633f8fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:31.655968 11144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 342c1044c2914cb6b95d9d37c6ec9f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:31.666649 11358 tablet_bootstrap.cc:492] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: Bootstrap starting.
I20260929 16:02:31.670128 11359 tablet_bootstrap.cc:492] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: Bootstrap starting.
I20260929 16:02:31.670500 11358 tablet_bootstrap.cc:654] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:31.671923 11358 log.cc:824] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:31.672951 11360 tablet_bootstrap.cc:492] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: Bootstrap starting.
I20260929 16:02:31.674744 11359 tablet_bootstrap.cc:654] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:31.676373 11358 tablet_bootstrap.cc:492] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: No bootstrap required, opened a new log
I20260929 16:02:31.676698 11358 ts_tablet_manager.cc:1397] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 16:02:31.676961 11360 tablet_bootstrap.cc:654] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:31.677003 11359 log.cc:824] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:31.677945 11360 log.cc:824] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:31.682286 11359 tablet_bootstrap.cc:492] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: No bootstrap required, opened a new log
I20260929 16:02:31.682284 11360 tablet_bootstrap.cc:492] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: No bootstrap required, opened a new log
I20260929 16:02:31.682610 11359 ts_tablet_manager.cc:1397] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260929 16:02:31.682610 11360 ts_tablet_manager.cc:1397] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:02:31.687022 11358 raft_consensus.cc:348] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:31.687534 11358 raft_consensus.cc:374] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:31.687672 11358 raft_consensus.cc:729] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b13a2b746c94d2ca553b7b582a49f4d, State: Initialized, Role: FOLLOWER
I20260929 16:02:31.688372 11358 consensus_queue.cc:260] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:31.689584 11340 heartbeater.cc:503] Master 127.23.222.126:46759 was elected leader, sending a full tablet report...
I20260929 16:02:31.690333 11358 ts_tablet_manager.cc:1428] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 16:02:31.694176 11359 raft_consensus.cc:348] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:31.694173 11360 raft_consensus.cc:348] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:31.694677 11359 raft_consensus.cc:374] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:31.694676 11360 raft_consensus.cc:374] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:31.694785 11359 raft_consensus.cc:729] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee423a846f54282a830bcd9c09a1f7d, State: Initialized, Role: FOLLOWER
I20260929 16:02:31.694785 11360 raft_consensus.cc:729] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd503c29398f4ba8b8775b2ab1a188f8, State: Initialized, Role: FOLLOWER
I20260929 16:02:31.695513 11359 consensus_queue.cc:260] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:31.695513 11360 consensus_queue.cc:260] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:31.698418 11209 heartbeater.cc:503] Master 127.23.222.126:46759 was elected leader, sending a full tablet report...
I20260929 16:02:31.698868 11359 ts_tablet_manager.cc:1428] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260929 16:02:31.699085 11360 ts_tablet_manager.cc:1428] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
W20260929 16:02:31.811560 11341 tablet.cc:2369] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:31.813279 11210 tablet.cc:2369] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:31.830327 11079 tablet.cc:2369] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:31.981673 11366 raft_consensus.cc:482] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:31.982146 11366 raft_consensus.cc:504] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:31.984001 11366 leader_election.cc:288] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b13a2b746c94d2ca553b7b582a49f4d (127.23.222.67:43385), cd503c29398f4ba8b8775b2ab1a188f8 (127.23.222.66:35315)
I20260929 16:02:31.990867 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.984465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:35059 (local address 127.23.222.67:43385)
0929 16:02:31.984741 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.984747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:31.985056 (+ 309us) server_negotiation.cc:407] Connection header received
0929 16:02:31.985300 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:31.985307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:31.985384 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:31.985485 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:31.986148 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.987170 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.989111 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.989941 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.990139 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.990238 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:31.990483 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 16:02:31.990608 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 16:02:31.990921 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.984339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:43385 (local address 127.23.222.65:35059)
0929 16:02:31.984938 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.984955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:31.985136 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:31.985508 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:31.985513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:31.985538 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:31.986008 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.986022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.987328 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.987333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.988918 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.988931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.990137 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.990143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.990315 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.990340 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:02:31.990444 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 16:02:31.990578 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":305,"threads_started":1}
I20260929 16:02:31.991645 11365 raft_consensus.cc:482] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:31.991770 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.985168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35315 (local address 127.23.222.65:60919)
0929 16:02:31.986250 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.986263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:31.986404 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:31.986770 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:31.986774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:31.986795 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:31.987159 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.987168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.988268 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.988275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.989607 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.989620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.990925 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.990937 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:31.991161 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.991191 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 16:02:31.991316 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 16:02:31.991448 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":790,"threads_started":1}
I20260929 16:02:31.991830 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.985479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:60919 (local address 127.23.222.66:35315)
0929 16:02:31.985718 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.985726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:31.986357 (+ 631us) server_negotiation.cc:407] Connection header received
0929 16:02:31.986555 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:31.986560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:31.986658 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:31.986755 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:31.987336 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.988137 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.989797 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.990596 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.990751 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:31.990797 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:31.991373 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 16:02:31.991502 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:02:31.992306 11295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342c1044c2914cb6b95d9d37c6ec9f23" candidate_uuid: "dee423a846f54282a830bcd9c09a1f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" is_pre_election: true
I20260929 16:02:31.992889 11295 raft_consensus.cc:2454] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee423a846f54282a830bcd9c09a1f7d in term 0.
I20260929 16:02:31.992888 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342c1044c2914cb6b95d9d37c6ec9f23" candidate_uuid: "dee423a846f54282a830bcd9c09a1f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" is_pre_election: true
I20260929 16:02:31.992014 11365 raft_consensus.cc:504] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:31.993553 10967 leader_election.cc:302] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b13a2b746c94d2ca553b7b582a49f4d, dee423a846f54282a830bcd9c09a1f7d; no voters:
I20260929 16:02:31.994031 11366 raft_consensus.cc:2781] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:31.994122 11164 raft_consensus.cc:2454] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee423a846f54282a830bcd9c09a1f7d in term 0.
I20260929 16:02:31.994202 11366 raft_consensus.cc:482] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:31.994303 11366 raft_consensus.cc:3037] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:31.995455 11365 leader_election.cc:288] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b13a2b746c94d2ca553b7b582a49f4d (127.23.222.67:43385), dee423a846f54282a830bcd9c09a1f7d (127.23.222.65:45633)
I20260929 16:02:31.998751 11366 raft_consensus.cc:504] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:31.999830 11366 leader_election.cc:288] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [CANDIDATE]: Term 1 election: Requested vote from peers 7b13a2b746c94d2ca553b7b582a49f4d (127.23.222.67:43385), cd503c29398f4ba8b8775b2ab1a188f8 (127.23.222.66:35315)
I20260929 16:02:32.000252 11295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342c1044c2914cb6b95d9d37c6ec9f23" candidate_uuid: "dee423a846f54282a830bcd9c09a1f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b13a2b746c94d2ca553b7b582a49f4d"
I20260929 16:02:32.000545 11295 raft_consensus.cc:3037] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:32.003106 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.995651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:43385 (local address 127.23.222.66:34069)
0929 16:02:31.996392 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:31.996413 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:31.996549 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:31.996877 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:31.996881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:31.996905 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:31.997317 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:31.997330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.998419 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:31.998423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:32.000313 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:32.000328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.002290 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.002295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:32.002471 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:32.002504 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 16:02:32.002603 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:02:32.002745 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":1469,"threads_started":1}
I20260929 16:02:32.003523 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.996171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:34069 (local address 127.23.222.67:43385)
0929 16:02:31.996574 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.996581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:31.996603 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:02:31.996677 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:31.996682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:31.996748 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:31.996852 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:31.997487 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:31.998275 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.000541 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.001851 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.001999 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:32.002075 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:32.002684 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 16:02:32.002923 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260929 16:02:32.004436 11294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342c1044c2914cb6b95d9d37c6ec9f23" candidate_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" is_pre_election: true
I20260929 16:02:32.004951 11295 raft_consensus.cc:2454] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee423a846f54282a830bcd9c09a1f7d in term 1.
I20260929 16:02:32.006031 10967 leader_election.cc:302] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b13a2b746c94d2ca553b7b582a49f4d, dee423a846f54282a830bcd9c09a1f7d; no voters:
I20260929 16:02:32.006493 11366 raft_consensus.cc:2781] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:32.007120 11366 raft_consensus.cc:686] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 1 LEADER]: Becoming Leader. State: Replica: dee423a846f54282a830bcd9c09a1f7d, State: Running, Role: LEADER
I20260929 16:02:32.007879 11164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342c1044c2914cb6b95d9d37c6ec9f23" candidate_uuid: "dee423a846f54282a830bcd9c09a1f7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd503c29398f4ba8b8775b2ab1a188f8"
I20260929 16:02:32.008139 11164 raft_consensus.cc:3037] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:32.008279 11366 consensus_queue.cc:237] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } }
I20260929 16:02:32.010721 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.996383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:45065 (local address 127.23.222.65:45633)
0929 16:02:31.997123 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:31.997130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:32.003471 (+ 6341us) server_negotiation.cc:407] Connection header received
0929 16:02:32.003715 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:32.003721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:32.003834 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:32.003902 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:32.004610 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.005748 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.008354 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.009513 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.009665 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:32.009768 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:32.010310 (+ 542us) server_negotiation.cc:299] Negotiation successful
0929 16:02:32.010446 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589}
I20260929 16:02:32.010782 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:31.995838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45633 (local address 127.23.222.66:45065)
0929 16:02:32.003378 (+ 7540us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.003398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:32.003568 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:32.004096 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:32.004100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:32.004127 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:32.004309 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:32.004321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.006057 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.006065 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:32.008099 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:32.008120 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.009911 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.009916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:32.010087 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:32.010115 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 16:02:32.010246 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 16:02:32.010415 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7414,"thread_start_us":11422,"threads_started":1}
I20260929 16:02:32.012225 11164 raft_consensus.cc:2454] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee423a846f54282a830bcd9c09a1f7d in term 1.
I20260929 16:02:32.014243 11032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "342c1044c2914cb6b95d9d37c6ec9f23" candidate_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dee423a846f54282a830bcd9c09a1f7d" is_pre_election: true
I20260929 16:02:32.015614 11097 leader_election.cc:302] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd503c29398f4ba8b8775b2ab1a188f8; no voters: 7b13a2b746c94d2ca553b7b582a49f4d, dee423a846f54282a830bcd9c09a1f7d
I20260929 16:02:32.016098 11365 raft_consensus.cc:2726] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 16:02:32.016330 10889 catalog_manager.cc:5495] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d reported cstate change: term changed from 0 to 1, leader changed from <none> to dee423a846f54282a830bcd9c09a1f7d (127.23.222.65). New cstate: current_term: 1 leader_uuid: "dee423a846f54282a830bcd9c09a1f7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dee423a846f54282a830bcd9c09a1f7d" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 } health_report { overall_health: UNKNOWN } } }
I20260929 16:02:32.116557 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.110298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.23.222.65:45633)
0929 16:02:32.110526 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:32.110536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:32.110982 (+ 446us) server_negotiation.cc:407] Connection header received
0929 16:02:32.111176 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:32.111180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:32.111341 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:32.111476 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:32.111960 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.112709 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.114189 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.114609 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.114700 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:32.115286 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:32.115822 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:32.115985 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:32.116156 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 16:02:32.116224 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 16:02:32.125191 11295 raft_consensus.cc:1261] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [term 1 FOLLOWER]: Refusing update from remote peer dee423a846f54282a830bcd9c09a1f7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:32.125252 11164 raft_consensus.cc:1261] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 1 FOLLOWER]: Refusing update from remote peer dee423a846f54282a830bcd9c09a1f7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:32.126170 11371 consensus_queue.cc:1035] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b13a2b746c94d2ca553b7b582a49f4d" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:32.126608 11366 consensus_queue.cc:1035] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd503c29398f4ba8b8775b2ab1a188f8" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:32.160719 10889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:32.161015 10889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:32.164498 10889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a245234bdcff489188ddfe75516c08c1: Sending DeleteTablet for 3 replicas of tablet 342c1044c2914cb6b95d9d37c6ec9f23
I20260929 16:02:32.165869 11144 tablet_service.cc:1514] Processing DeleteTablet for tablet 342c1044c2914cb6b95d9d37c6ec9f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:32 UTC) from {username='slave'} at 127.0.0.1:41594
I20260929 16:02:32.165879 11012 tablet_service.cc:1514] Processing DeleteTablet for tablet 342c1044c2914cb6b95d9d37c6ec9f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:32 UTC) from {username='slave'} at 127.0.0.1:38440
I20260929 16:02:32.166529 11275 tablet_service.cc:1514] Processing DeleteTablet for tablet 342c1044c2914cb6b95d9d37c6ec9f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:32 UTC) from {username='slave'} at 127.0.0.1:35776
I20260929 16:02:32.167161 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 10950
I20260929 16:02:32.191715 11388 tablet_replica.cc:330] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: stopping tablet replica
I20260929 16:02:32.192242 11388 raft_consensus.cc:2229] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:32.192866 11388 raft_consensus.cc:2258] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:32.195065 11388 ts_tablet_manager.cc:1905] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:32.203111 11388 ts_tablet_manager.cc:1918] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:32.203349 11388 log.cc:1196] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/342c1044c2914cb6b95d9d37c6ec9f23
I20260929 16:02:32.203895 11388 ts_tablet_manager.cc:1939] T 342c1044c2914cb6b95d9d37c6ec9f23 P 7b13a2b746c94d2ca553b7b582a49f4d: Deleting consensus metadata
I20260929 16:02:32.204383 11386 tablet_replica.cc:330] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: stopping tablet replica
I20260929 16:02:32.205000 11386 raft_consensus.cc:2229] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:32.205787 11386 raft_consensus.cc:2258] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:32.206012 10876 catalog_manager.cc:4841] TS 7b13a2b746c94d2ca553b7b582a49f4d (127.23.222.67:43385): tablet 342c1044c2914cb6b95d9d37c6ec9f23 (table test-table [id=5bc3f5c2f82046079b490991f633f8fe]) successfully deleted
I20260929 16:02:32.206633 11387 tablet_replica.cc:330] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: stopping tablet replica
I20260929 16:02:32.207520 11387 raft_consensus.cc:2229] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:32.208454 11387 raft_consensus.cc:2258] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:32.208489 11386 ts_tablet_manager.cc:1905] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:32.210107 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.168588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.23.222.65:45633)
0929 16:02:32.168836 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:32.168845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:32.188450 (+ 19605us) server_negotiation.cc:407] Connection header received
0929 16:02:32.188586 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:32.188595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:32.188781 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:32.188938 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:32.189917 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.191427 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.194255 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.198142 (+ 3887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.198298 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:32.208301 (+ 10003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:32.208314 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:32.208331 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:32.209136 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:32.209225 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:32.209253 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:32.209375 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:32.209571 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 16:02:32.209715 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 16:02:32.210853 11072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:32.210968 11387 ts_tablet_manager.cc:1905] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:32.216487 11387 ts_tablet_manager.cc:1918] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:32.216643 11387 log.cc:1196] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/342c1044c2914cb6b95d9d37c6ec9f23
I20260929 16:02:32.217161 11387 ts_tablet_manager.cc:1939] T 342c1044c2914cb6b95d9d37c6ec9f23 P cd503c29398f4ba8b8775b2ab1a188f8: Deleting consensus metadata
I20260929 16:02:32.219182 10876 catalog_manager.cc:4841] TS cd503c29398f4ba8b8775b2ab1a188f8 (127.23.222.66:35315): tablet 342c1044c2914cb6b95d9d37c6ec9f23 (table test-table [id=5bc3f5c2f82046079b490991f633f8fe]) successfully deleted
I20260929 16:02:32.317107 11386 ts_tablet_manager.cc:1918] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:32.317276 11386 log.cc:1196] T 342c1044c2914cb6b95d9d37c6ec9f23 P dee423a846f54282a830bcd9c09a1f7d: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/342c1044c2914cb6b95d9d37c6ec9f23
I20260929 16:02:32.317497 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 10950
W20260929 16:02:32.336265 10874 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.222.65:45633 (error 108)
I20260929 16:02:32.337015 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 11082
I20260929 16:02:32.338384 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.337551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45633 (local address 127.0.0.1:59166)
0929 16:02:32.338095 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.338206 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":301,"threads_started":1}
W20260929 16:02:32.338618 10874 catalog_manager.cc:4584] TS dee423a846f54282a830bcd9c09a1f7d (127.23.222.65:45633): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 342c1044c2914cb6b95d9d37c6ec9f23: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45633: connect: Connection refused (error 111)
I20260929 16:02:32.344383 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.337845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55894 (local address 127.23.222.66:35315)
0929 16:02:32.338035 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:32.338041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:32.338145 (+ 104us) server_negotiation.cc:407] Connection header received
0929 16:02:32.338363 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:32.338367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:32.338484 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:32.338582 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:32.339085 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.339881 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.341179 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.341613 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.341710 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:32.342777 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:32.342786 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:32.342800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:32.343520 (+ 720us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:32.343571 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:32.343592 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:32.343687 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:32.343843 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 16:02:32.343940 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:02:32.345189 11204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:32.392383 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.391864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45633 (local address 127.0.0.1:59172)
0929 16:02:32.392089 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.392200 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:02:32.393581 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.393166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45633 (local address 127.0.0.1:59184)
0929 16:02:32.393329 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.393409 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 16:02:32.436070 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 11082
I20260929 16:02:32.454094 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 11212
I20260929 16:02:32.454176 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.453694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45633 (local address 127.0.0.1:59196)
0929 16:02:32.453930 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.454021 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:02:32.455427 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.454882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45633 (local address 127.0.0.1:59210)
0929 16:02:32.455162 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.455228 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260929 16:02:32.461477 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.455009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.23.222.67:43385)
0929 16:02:32.455188 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:32.455194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:32.455213 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:02:32.455396 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:32.455401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:32.455546 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:32.455648 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:32.456068 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.456813 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.458362 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:32.458787 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:32.458940 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:32.460162 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:32.460171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:32.460186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:32.460749 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:32.460797 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:32.460816 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:32.460964 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:32.461072 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 16:02:32.461177 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 16:02:32.462201 11335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:32.536228 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.535688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45633 (local address 127.0.0.1:59224)
0929 16:02:32.535971 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.536069 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 16:02:32.537338 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:32.536917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45633 (local address 127.0.0.1:59238)
0929 16:02:32.537136 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:32.537210 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 16:02:32.552707 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 11212
I20260929 16:02:32.570006 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 10859
I20260929 16:02:32.571226 10920 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:32.683534 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 10859
2026-09-29T16:02:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3286 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:32 dist-test-slave-cq88 krb5kdc[11398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:32 dist-test-slave-cq88 krb5kdc[11398](info): set up 2 sockets
Sep 29 16:02:32 dist-test-slave-cq88 krb5kdc[11398](info): commencing operation
krb5kdc: starting...
W20260929 16:02:33.854507 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:34 dist-test-slave-cq88 krb5kdc[11398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:02:34.123175 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34425
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:34.471074 11411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:34.471426 11411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:34.471582 11411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:34.478787 11411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:34.478891 11411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:34.478995 11411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:34.479102 11411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:34.479183 11411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:34.489319 11411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:34.490665 11411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:34.492097 11411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:34.504657 11419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:34.504747 11417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:34.504998 11416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:34.505820 11411 server_base.cc:1034] running on GCE node
I20260929 16:02:34.506675 11411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:34.508335 11411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:34.509579 11411 hybrid_clock.cc:648] HybridClock initialized: now 1790697754509512 us; error 54 us; skew 500 ppm
I20260929 16:02:34.510039 11411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:34 dist-test-slave-cq88 krb5kdc[11398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697754, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:34.513531 11411 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:02:34.515182 11411 webserver.cc:460] Webserver started at http://127.23.222.126:41579/ using document root <none> and password file <none>
I20260929 16:02:34.515928 11411 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:34.516091 11411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:34.516412 11411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:34.519586 11411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "3894784c8fc24a2f8f53037839f71d55"
format_stamp: "Formatted at 2026-09-29 16:02:34 on dist-test-slave-cq88"
server_key: "29079a5b2bc2d38f4d4d6552272e3e78"
server_key_iv: "453890cc9bacdff016b71a27e10241cb"
server_key_version: "encryptionkey@0"
I20260929 16:02:34.520371 11411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "3894784c8fc24a2f8f53037839f71d55"
format_stamp: "Formatted at 2026-09-29 16:02:34 on dist-test-slave-cq88"
server_key: "29079a5b2bc2d38f4d4d6552272e3e78"
server_key_iv: "453890cc9bacdff016b71a27e10241cb"
server_key_version: "encryptionkey@0"
I20260929 16:02:34.525682 11411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 16:02:34.529203 11426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:34.530540 11411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:02:34.530720 11411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "3894784c8fc24a2f8f53037839f71d55"
format_stamp: "Formatted at 2026-09-29 16:02:34 on dist-test-slave-cq88"
server_key: "29079a5b2bc2d38f4d4d6552272e3e78"
server_key_iv: "453890cc9bacdff016b71a27e10241cb"
server_key_version: "encryptionkey@0"
I20260929 16:02:34.530923 11411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:34.545977 11411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:34.550521 11411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:34.550838 11411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:34.570250 11411 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:34425
I20260929 16:02:34.570281 11478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:34425 every 8 connection(s)
I20260929 16:02:34.571759 11411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:34.575642 11479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:34.578578 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 11411
I20260929 16:02:34.578769 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:34.579329 24441 external_mini_cluster.cc:1428] Setting key 032db07101e8f9a567674f780d041452
I20260929 16:02:34.588600 11479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55: Bootstrap starting.
I20260929 16:02:34.592613 11479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:34.594058 11479 log.cc:824] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55: Log is configured to *not* fsync() on all Append() calls
Sep 29 16:02:34 dist-test-slave-cq88 krb5kdc[11398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:34.597643 11479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55: No bootstrap required, opened a new log
I20260929 16:02:34.604708 11482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:34.581430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33508 (local address 127.23.222.126:34425)
0929 16:02:34.582324 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:34.582336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:34.582379 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:02:34.584214 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:34.584240 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:34.584690 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:34.585031 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:34.586982 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:34.588470 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:34.590308 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:34.591026 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:34.591162 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:34.596854 (+ 5692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:34.596864 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:34.596880 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:34.596953 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:34.600318 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:34.601197 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:34.601203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:34.601210 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:34.601378 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:34.601791 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:34.601794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:34.601796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:34.602351 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:34.602590 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:34.603067 (+ 477us) server_negotiation.cc:299] Negotiation successful
0929 16:02:34.603587 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":451,"threads_started":1}
I20260929 16:02:34.607205 11479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3894784c8fc24a2f8f53037839f71d55" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34425 } }
I20260929 16:02:34.607702 11479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:34.607847 11479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3894784c8fc24a2f8f53037839f71d55, State: Initialized, Role: FOLLOWER
I20260929 16:02:34.608465 11479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3894784c8fc24a2f8f53037839f71d55" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34425 } }
I20260929 16:02:34.608705 11479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:34.608846 11479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:34.609015 11479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:34.611307 11479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3894784c8fc24a2f8f53037839f71d55" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34425 } }
I20260929 16:02:34.611835 11479 leader_election.cc:302] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3894784c8fc24a2f8f53037839f71d55; no voters:
I20260929 16:02:34.612429 11479 leader_election.cc:288] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:34.612576 11484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:34.613274 11484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [term 1 LEADER]: Becoming Leader. State: Replica: 3894784c8fc24a2f8f53037839f71d55, State: Running, Role: LEADER
I20260929 16:02:34.613998 11484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3894784c8fc24a2f8f53037839f71d55" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34425 } }
I20260929 16:02:34.614795 11479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:34.617185 11485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3894784c8fc24a2f8f53037839f71d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3894784c8fc24a2f8f53037839f71d55" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34425 } } }
I20260929 16:02:34.617223 11486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3894784c8fc24a2f8f53037839f71d55. Latest consensus state: current_term: 1 leader_uuid: "3894784c8fc24a2f8f53037839f71d55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3894784c8fc24a2f8f53037839f71d55" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 34425 } } }
I20260929 16:02:34.617587 11485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:34.617625 11486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:34.622709 11491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:34.628528 11491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:34.639983 11491 catalog_manager.cc:1347] Generated new cluster ID: 27a382a071fc4134b3b636527b7c6905
I20260929 16:02:34.640116 11491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:34.656762 11491 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:34.658244 11491 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:34.666375 11491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55: Generated new TSK 0
I20260929 16:02:34.667400 11491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.222.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
Entry for principal kudu/127.23.222.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab.
I20260929 16:02:34.742910 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34425
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:35.082361 11505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:35.082685 11505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:35.082857 11505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:35.090029 11505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:35.090183 11505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:35.090372 11505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:02:35.100330 11505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.65.keytab
--principal=kudu/127.23.222.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:35.101714 11505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:35.103070 11505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:35.118257 11511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:35.118233 11513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:35.118388 11510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:35.119078 11505 server_base.cc:1034] running on GCE node
I20260929 16:02:35.119858 11505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:35.120929 11505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:35.122181 11505 hybrid_clock.cc:648] HybridClock initialized: now 1790697755122102 us; error 81 us; skew 500 ppm
I20260929 16:02:35.122529 11505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:35 dist-test-slave-cq88 krb5kdc[11398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697755, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:35.125942 11505 init.cc:377] Logged in from keytab as kudu/127.23.222.65@KRBTEST.COM (short username kudu)
I20260929 16:02:35.127681 11505 webserver.cc:460] Webserver started at http://127.23.222.65:42361/ using document root <none> and password file <none>
I20260929 16:02:35.128454 11505 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:35.128614 11505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:35.128949 11505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:35.132233 11505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "6bb19c0418814cc0aa842e53a10d3209"
format_stamp: "Formatted at 2026-09-29 16:02:35 on dist-test-slave-cq88"
server_key: "1c778ab1fb52bdbe570ce1151c6203ce"
server_key_iv: "f78bf1466265f0b896e48e61f8a37cb6"
server_key_version: "encryptionkey@0"
I20260929 16:02:35.133049 11505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "6bb19c0418814cc0aa842e53a10d3209"
format_stamp: "Formatted at 2026-09-29 16:02:35 on dist-test-slave-cq88"
server_key: "1c778ab1fb52bdbe570ce1151c6203ce"
server_key_iv: "f78bf1466265f0b896e48e61f8a37cb6"
server_key_version: "encryptionkey@0"
I20260929 16:02:35.138829 11505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:02:35.142460 11520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:35.143894 11505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:35.144083 11505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "6bb19c0418814cc0aa842e53a10d3209"
format_stamp: "Formatted at 2026-09-29 16:02:35 on dist-test-slave-cq88"
server_key: "1c778ab1fb52bdbe570ce1151c6203ce"
server_key_iv: "f78bf1466265f0b896e48e61f8a37cb6"
server_key_version: "encryptionkey@0"
I20260929 16:02:35.144289 11505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:35.160662 11505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:35.165127 11505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:35.165431 11505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:35.166625 11505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:35.168031 11505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:35.168140 11505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:35.168290 11505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:35.168362 11505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:35.201953 11505 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:40021
I20260929 16:02:35.202003 11633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:40021 every 8 connection(s)
I20260929 16:02:35.203714 11505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:35.213549 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 11505
I20260929 16:02:35.213764 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:35.214218 24441 external_mini_cluster.cc:1428] Setting key 365da09bd17897947d26cb3f364829e4
Sep 29 16:02:35 dist-test-slave-cq88 krb5kdc[11398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697755, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.65@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:35.225103 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:35.207422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:53483 (local address 127.23.222.126:34425)
0929 16:02:35.207882 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:35.207887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:35.208820 (+ 933us) server_negotiation.cc:407] Connection header received
0929 16:02:35.210619 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:35.210625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:35.210717 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:35.210837 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:35.212440 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:35.213163 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:35.215272 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:35.215755 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:35.215899 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:35.219314 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:35.219318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:35.219322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:35.219353 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:35.221828 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:35.222543 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:35.222550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:35.222554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:35.222680 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:35.223129 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:35.223135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:35.223139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:35.223489 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:35.223657 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:35.224550 (+ 893us) server_negotiation.cc:299] Negotiation successful
0929 16:02:35.224720 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":242,"threads_started":1}
I20260929 16:02:35.226179 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:35.207951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34425 (local address 127.23.222.65:53483)
0929 16:02:35.208659 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:35.208699 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:35.210365 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:35.211078 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:35.211086 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:35.211699 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:35.212268 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:35.212285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:35.213411 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:35.213419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:35.215098 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:35.215111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:35.215987 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:35.215997 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:35.216273 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:35.216957 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:35.216980 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:35.219084 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:35.221967 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:35.221982 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:35.221998 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:35.222422 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:35.222796 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:35.222801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:35.222805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:35.223015 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:35.223657 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:35.223666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:35.224136 (+ 470us) client_negotiation.cc:769] Sending connection context
0929 16:02:35.224480 (+ 344us) client_negotiation.cc:240] Negotiation successful
0929 16:02:35.225046 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":265,"threads_started":1}
I20260929 16:02:35.227757 11634 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34425
I20260929 16:02:35.228077 11634 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:35.228797 11634 heartbeater.cc:511] Master 127.23.222.126:34425 requested a full tablet report, sending...
I20260929 16:02:35.230834 11443 ts_manager.cc:194] Registered new tserver with Master: 6bb19c0418814cc0aa842e53a10d3209 (127.23.222.65:40021)
I20260929 16:02:35.232342 11443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.65@KRBTEST.COM'} at 127.23.222.65:53483
WARNING: no policy specified for kudu/127.23.222.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
Entry for principal kudu/127.23.222.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab.
I20260929 16:02:35.270870 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34425
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:35.601449 11640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:35.601809 11640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:35.601994 11640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:35.609376 11640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:35.609571 11640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:35.609761 11640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:02:35.620157 11640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.66.keytab
--principal=kudu/127.23.222.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:35.621536 11640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:35.622941 11640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:35.636768 11645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:35.636945 11648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:35.636902 11646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:35.637925 11640 server_base.cc:1034] running on GCE node
I20260929 16:02:35.638710 11640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:35.639808 11640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:35.641047 11640 hybrid_clock.cc:648] HybridClock initialized: now 1790697755640999 us; error 59 us; skew 500 ppm
I20260929 16:02:35.641392 11640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:35 dist-test-slave-cq88 krb5kdc[11398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697755, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:35.644593 11640 init.cc:377] Logged in from keytab as kudu/127.23.222.66@KRBTEST.COM (short username kudu)
I20260929 16:02:35.646044 11640 webserver.cc:460] Webserver started at http://127.23.222.66:33615/ using document root <none> and password file <none>
I20260929 16:02:35.646733 11640 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:35.646893 11640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:35.647212 11640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:35.650431 11640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "3877e25dd7574a54a1712af376dd9f5a"
format_stamp: "Formatted at 2026-09-29 16:02:35 on dist-test-slave-cq88"
server_key: "e4e0085ff61918ca8dc58c0d9de00a22"
server_key_iv: "53d63654b21e63be299e734c6c1d60c5"
server_key_version: "encryptionkey@0"
I20260929 16:02:35.651216 11640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "3877e25dd7574a54a1712af376dd9f5a"
format_stamp: "Formatted at 2026-09-29 16:02:35 on dist-test-slave-cq88"
server_key: "e4e0085ff61918ca8dc58c0d9de00a22"
server_key_iv: "53d63654b21e63be299e734c6c1d60c5"
server_key_version: "encryptionkey@0"
I20260929 16:02:35.656431 11640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:02:35.659812 11655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:35.661115 11640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:35.661296 11640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "3877e25dd7574a54a1712af376dd9f5a"
format_stamp: "Formatted at 2026-09-29 16:02:35 on dist-test-slave-cq88"
server_key: "e4e0085ff61918ca8dc58c0d9de00a22"
server_key_iv: "53d63654b21e63be299e734c6c1d60c5"
server_key_version: "encryptionkey@0"
I20260929 16:02:35.661504 11640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:35.708329 11640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:35.712949 11640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:35.713269 11640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:35.714412 11640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:35.716529 11640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:35.716626 11640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:35.716753 11640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:35.716830 11640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:35.751096 11640 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:35745
I20260929 16:02:35.751137 11768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:35745 every 8 connection(s)
I20260929 16:02:35.752498 11640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:35.756574 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 11640
I20260929 16:02:35.756778 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:35.757285 24441 external_mini_cluster.cc:1428] Setting key ceca2275dc3332e0a7efa627b7ca2008
Sep 29 16:02:35 dist-test-slave-cq88 krb5kdc[11398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697755, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.66@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:35.775605 11771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:35.755210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:42205 (local address 127.23.222.126:34425)
0929 16:02:35.755746 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:35.755751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:35.757816 (+ 2065us) server_negotiation.cc:407] Connection header received
0929 16:02:35.758933 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:35.758938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:35.759026 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:35.759144 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:35.760791 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:35.761537 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:35.763104 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:35.763603 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:35.763734 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:35.768510 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:35.768520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:35.768528 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:35.768575 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:35.772162 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:35.772911 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:35.772919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:35.772926 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:35.773093 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:35.773520 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:35.773528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:35.773534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:35.773907 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:35.774069 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:35.774770 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 16:02:35.775063 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":224,"threads_started":1}
I20260929 16:02:35.776332 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:35.755871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34425 (local address 127.23.222.66:42205)
0929 16:02:35.756597 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:35.756634 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:35.758651 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:35.759419 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:35.759429 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:35.760046 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:35.760610 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:35.760627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:35.761701 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:35.761705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:35.762953 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:35.762965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:35.763842 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:35.763849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:35.764087 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:35.764803 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:35.764845 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:35.768203 (+ 3358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:35.772339 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:35.772345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:35.772358 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:35.772763 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:35.773205 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:35.773208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:35.773210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:35.773401 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:35.774040 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:35.774045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:35.774395 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 16:02:35.774623 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 16:02:35.775050 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":310,"threads_started":1}
I20260929 16:02:35.777854 11769 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34425
I20260929 16:02:35.778215 11769 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:35.779116 11769 heartbeater.cc:511] Master 127.23.222.126:34425 requested a full tablet report, sending...
I20260929 16:02:35.780869 11443 ts_manager.cc:194] Registered new tserver with Master: 3877e25dd7574a54a1712af376dd9f5a (127.23.222.66:35745)
I20260929 16:02:35.781725 11443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.66@KRBTEST.COM'} at 127.23.222.66:42205
WARNING: no policy specified for kudu/127.23.222.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
Entry for principal kudu/127.23.222.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab.
I20260929 16:02:35.817102 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34425
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:44951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:36.152915 11775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:36.153215 11775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:36.153326 11775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:36.161171 11775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:36.161291 11775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:36.161418 11775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:02:36.171852 11775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:44951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.67.keytab
--principal=kudu/127.23.222.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:34425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:36.173141 11775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:36.174417 11775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:36.191076 11781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:36.191116 11784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:36.191171 11782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:36.191959 11775 server_base.cc:1034] running on GCE node
I20260929 16:02:36.192615 11775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:36.193585 11775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:36.194818 11775 hybrid_clock.cc:648] HybridClock initialized: now 1790697756194764 us; error 48 us; skew 500 ppm
I20260929 16:02:36.195200 11775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:36 dist-test-slave-cq88 krb5kdc[11398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697756, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:36.198314 11775 init.cc:377] Logged in from keytab as kudu/127.23.222.67@KRBTEST.COM (short username kudu)
I20260929 16:02:36.199780 11775 webserver.cc:460] Webserver started at http://127.23.222.67:34117/ using document root <none> and password file <none>
I20260929 16:02:36.200414 11775 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:36.200525 11775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:36.200788 11775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:36.203783 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "432b8f586d7743d685b0c3095f64e1e9"
format_stamp: "Formatted at 2026-09-29 16:02:36 on dist-test-slave-cq88"
server_key: "cd338e2a33606c0e1bfb69a278b8d450"
server_key_iv: "90b79abefc4501c6c0f5d833317f2e81"
server_key_version: "encryptionkey@0"
I20260929 16:02:36.204559 11775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "432b8f586d7743d685b0c3095f64e1e9"
format_stamp: "Formatted at 2026-09-29 16:02:36 on dist-test-slave-cq88"
server_key: "cd338e2a33606c0e1bfb69a278b8d450"
server_key_iv: "90b79abefc4501c6c0f5d833317f2e81"
server_key_version: "encryptionkey@0"
I20260929 16:02:36.209877 11775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:02:36.213511 11791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:36.214912 11775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:02:36.215099 11775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "432b8f586d7743d685b0c3095f64e1e9"
format_stamp: "Formatted at 2026-09-29 16:02:36 on dist-test-slave-cq88"
server_key: "cd338e2a33606c0e1bfb69a278b8d450"
server_key_iv: "90b79abefc4501c6c0f5d833317f2e81"
server_key_version: "encryptionkey@0"
I20260929 16:02:36.215390 11775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:36.230427 11775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:36.234890 11775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:36.235208 11775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:36.235821 11634 heartbeater.cc:503] Master 127.23.222.126:34425 was elected leader, sending a full tablet report...
I20260929 16:02:36.236492 11775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:36.237978 11775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:36.238077 11775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:36.238215 11775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:36.238293 11775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:36.272869 11775 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:39729
I20260929 16:02:36.272913 11904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:39729 every 8 connection(s)
I20260929 16:02:36.274216 11775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:36.282536 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 11775
I20260929 16:02:36.282722 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:36.283193 24441 external_mini_cluster.cc:1428] Setting key e719a400194a462431d143885292fe7a
Sep 29 16:02:36 dist-test-slave-cq88 krb5kdc[11398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697756, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.67@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:02:36.294214 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.276945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:51075 (local address 127.23.222.126:34425)
0929 16:02:36.277373 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.277377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.278385 (+ 1008us) server_negotiation.cc:407] Connection header received
0929 16:02:36.280328 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.280334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.280426 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.280560 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:36.282040 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.282941 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.284785 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.285347 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.285489 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.288905 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:36.288910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:36.288913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:36.288947 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:36.291499 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.292165 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.292168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.292170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.292282 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.292640 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.292643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.292645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.292894 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:36.293011 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.293633 (+ 622us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.293789 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":210,"threads_started":1}
I20260929 16:02:36.295010 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.277533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:34425 (local address 127.23.222.67:51075)
0929 16:02:36.278225 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:36.278268 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:36.280045 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:36.280792 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:36.280800 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:36.281307 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:36.281859 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.281876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.283097 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.283102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.284646 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.284659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.285548 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.285554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.285799 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.286563 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:02:36.286586 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:36.288708 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:36.291641 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:36.291647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:36.291659 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:36.292056 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:36.292370 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:02:36.292374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:02:36.292376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:02:36.292543 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:02:36.292995 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:36.293000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:36.293305 (+ 305us) client_negotiation.cc:769] Sending connection context
0929 16:02:36.293528 (+ 223us) client_negotiation.cc:240] Negotiation successful
0929 16:02:36.293906 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":267,"threads_started":1}
I20260929 16:02:36.296418 11905 heartbeater.cc:347] Connected to a master server at 127.23.222.126:34425
I20260929 16:02:36.296784 11905 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:36.297633 11905 heartbeater.cc:511] Master 127.23.222.126:34425 requested a full tablet report, sending...
I20260929 16:02:36.299106 11443 ts_manager.cc:194] Registered new tserver with Master: 432b8f586d7743d685b0c3095f64e1e9 (127.23.222.67:39729)
I20260929 16:02:36.299933 11443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.222.67@KRBTEST.COM'} at 127.23.222.67:51075
I20260929 16:02:36.310489 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:36.326999 11907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.314483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.23.222.126:34425)
0929 16:02:36.314648 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.314653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.315158 (+ 505us) server_negotiation.cc:407] Connection header received
0929 16:02:36.315374 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.315378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.315441 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.315543 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:36.316926 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.317639 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.318973 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.319416 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.319525 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.321816 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:36.321821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:36.321824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:36.321856 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:36.324705 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.325301 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.325305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.325307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.325418 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.325739 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.325742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.325744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.326061 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:36.326238 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.326524 (+ 286us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.326684 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:02:36.330936 11443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:36.338248 11443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:36.373232 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.350592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:39729 (local address 127.0.0.1:57416)
0929 16:02:36.362660 (+ 12068us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:36.362711 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:36.362924 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:36.366519 (+ 3595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:36.366527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:36.366554 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:36.366995 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.367008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.368778 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.368784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.370549 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.370562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.372060 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.372065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.372257 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.372312 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 16:02:36.372478 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 16:02:36.372697 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11813,"mutex_wait_us":25,"thread_start_us":11712,"threads_started":1}
I20260929 16:02:36.373459 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.353833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35088 (local address 127.23.222.66:35745)
0929 16:02:36.363767 (+ 9934us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.363776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.364952 (+ 1176us) server_negotiation.cc:407] Connection header received
0929 16:02:36.365278 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.365288 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.365532 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.365702 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:36.366619 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.368211 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.370871 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.372316 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.372471 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.372659 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.373006 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.373172 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9781,"thread_start_us":9687,"threads_started":1}
I20260929 16:02:36.373991 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.351189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.23.222.65:40021)
0929 16:02:36.362253 (+ 11064us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.362266 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.364318 (+ 2052us) server_negotiation.cc:407] Connection header received
0929 16:02:36.364535 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.364546 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.364840 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.365759 (+ 919us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:36.366068 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.367661 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.370521 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.372026 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.372165 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.372423 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.373323 (+ 900us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.373757 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10848,"thread_start_us":10642,"threads_started":1}
I20260929 16:02:36.374066 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.350785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.23.222.67:39729)
0929 16:02:36.365607 (+ 14822us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.365617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.365680 (+ 63us) server_negotiation.cc:407] Connection header received
0929 16:02:36.365774 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.365783 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.366045 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.366240 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:36.367201 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.368637 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.370741 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.371860 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.372016 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.372270 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.373496 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.373759 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14542,"thread_start_us":14825,"threads_started":1}
I20260929 16:02:36.373571 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.353227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35745 (local address 127.0.0.1:35088)
0929 16:02:36.364841 (+ 11614us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:36.364859 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:36.365058 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:36.365893 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:36.365899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:36.365927 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:36.366380 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.366393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.368359 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.368365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.370678 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.370691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.372577 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.372582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.372760 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.372824 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 16:02:36.372923 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 16:02:36.373056 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11505,"mutex_wait_us":25,"thread_start_us":10700,"threads_started":1}
I20260929 16:02:36.373862 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.350775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:40021 (local address 127.0.0.1:41968)
0929 16:02:36.364182 (+ 13407us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:36.364198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:36.364345 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:36.365097 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:36.365157 (+ 60us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:36.365194 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:36.365569 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.365584 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.367788 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.367795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.370310 (+ 2515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.370324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.372592 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.372598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.372814 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.372841 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 16:02:36.372949 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:02:36.373086 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13289,"mutex_wait_us":109,"thread_start_us":13815,"threads_started":1}
I20260929 16:02:36.377910 11704 tablet_service.cc:1467] Processing CreateTablet for tablet d3b84232b57a48e2ae41ee9b84a0513c (DEFAULT_TABLE table=test-table [id=bcbfce4e0615498eb3d1e91a9a676af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:36.377912 11569 tablet_service.cc:1467] Processing CreateTablet for tablet d3b84232b57a48e2ae41ee9b84a0513c (DEFAULT_TABLE table=test-table [id=bcbfce4e0615498eb3d1e91a9a676af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:36.379209 11704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b84232b57a48e2ae41ee9b84a0513c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:36.379318 11569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b84232b57a48e2ae41ee9b84a0513c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:36.381763 11839 tablet_service.cc:1467] Processing CreateTablet for tablet d3b84232b57a48e2ae41ee9b84a0513c (DEFAULT_TABLE table=test-table [id=bcbfce4e0615498eb3d1e91a9a676af8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:36.383280 11839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b84232b57a48e2ae41ee9b84a0513c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:36.391044 11924 tablet_bootstrap.cc:492] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: Bootstrap starting.
I20260929 16:02:36.394649 11924 tablet_bootstrap.cc:654] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:36.395987 11925 tablet_bootstrap.cc:492] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: Bootstrap starting.
I20260929 16:02:36.395962 11924 log.cc:824] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:36.398945 11927 tablet_bootstrap.cc:492] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: Bootstrap starting.
I20260929 16:02:36.399688 11925 tablet_bootstrap.cc:654] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:36.399715 11924 tablet_bootstrap.cc:492] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: No bootstrap required, opened a new log
I20260929 16:02:36.399997 11924 ts_tablet_manager.cc:1397] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 16:02:36.401137 11925 log.cc:824] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:36.402160 11927 tablet_bootstrap.cc:654] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:36.403477 11927 log.cc:824] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:36.404501 11925 tablet_bootstrap.cc:492] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: No bootstrap required, opened a new log
I20260929 16:02:36.404810 11925 ts_tablet_manager.cc:1397] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 16:02:36.406492 11927 tablet_bootstrap.cc:492] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: No bootstrap required, opened a new log
I20260929 16:02:36.406741 11927 ts_tablet_manager.cc:1397] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 16:02:36.408744 11924 raft_consensus.cc:348] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } }
I20260929 16:02:36.409145 11924 raft_consensus.cc:374] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:36.409253 11924 raft_consensus.cc:729] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3877e25dd7574a54a1712af376dd9f5a, State: Initialized, Role: FOLLOWER
I20260929 16:02:36.409885 11924 consensus_queue.cc:260] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } }
I20260929 16:02:36.411012 11769 heartbeater.cc:503] Master 127.23.222.126:34425 was elected leader, sending a full tablet report...
I20260929 16:02:36.411759 11924 ts_tablet_manager.cc:1428] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 16:02:36.415853 11925 raft_consensus.cc:348] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } }
I20260929 16:02:36.416343 11925 raft_consensus.cc:374] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:36.416458 11925 raft_consensus.cc:729] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bb19c0418814cc0aa842e53a10d3209, State: Initialized, Role: FOLLOWER
I20260929 16:02:36.417120 11925 consensus_queue.cc:260] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } }
I20260929 16:02:36.417484 11927 raft_consensus.cc:348] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } }
I20260929 16:02:36.417989 11927 raft_consensus.cc:374] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:36.418099 11927 raft_consensus.cc:729] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 432b8f586d7743d685b0c3095f64e1e9, State: Initialized, Role: FOLLOWER
I20260929 16:02:36.418749 11927 consensus_queue.cc:260] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } }
I20260929 16:02:36.419173 11925 ts_tablet_manager.cc:1428] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 16:02:36.420125 11905 heartbeater.cc:503] Master 127.23.222.126:34425 was elected leader, sending a full tablet report...
I20260929 16:02:36.420710 11927 ts_tablet_manager.cc:1428] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260929 16:02:36.456956 11635 tablet.cc:2369] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:36.486102 11931 raft_consensus.cc:482] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:36.486615 11931 raft_consensus.cc:504] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } }
I20260929 16:02:36.488771 11931 leader_election.cc:288] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 432b8f586d7743d685b0c3095f64e1e9 (127.23.222.67:39729), 3877e25dd7574a54a1712af376dd9f5a (127.23.222.66:35745)
I20260929 16:02:36.495996 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.489320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:36365 (local address 127.23.222.66:35745)
0929 16:02:36.489561 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.489568 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.490586 (+ 1018us) server_negotiation.cc:407] Connection header received
0929 16:02:36.490689 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.490699 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.490782 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.490896 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:36.491571 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.492257 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.494178 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.495003 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.495121 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.495174 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.495607 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.495739 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:02:36.496232 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.489211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:55975 (local address 127.23.222.67:39729)
0929 16:02:36.489681 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.489688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.490248 (+ 560us) server_negotiation.cc:407] Connection header received
0929 16:02:36.490519 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.490525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.490617 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.490744 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:36.491458 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.492244 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.494207 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.495271 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.495420 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.495496 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.495857 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.495986 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260929 16:02:36.496343 11934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.489283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:35745 (local address 127.23.222.65:36365)
0929 16:02:36.490383 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:36.490401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:36.490530 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:36.490930 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:36.490943 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:36.490977 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:36.491407 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.491423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.492444 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.492448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.494013 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.494026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.495215 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.495222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.495422 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.495459 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 16:02:36.495564 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 16:02:36.495664 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":27,"thread_start_us":649,"threads_started":1}
I20260929 16:02:36.496387 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.489010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:39729 (local address 127.23.222.65:55975)
0929 16:02:36.490133 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:36.490152 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:36.490303 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:36.490747 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:36.490752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:36.490779 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:36.491290 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.491307 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.492376 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.492384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.494026 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:36.494037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.495491 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.495495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:36.495677 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.495709 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 16:02:36.495806 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 16:02:36.495942 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":96,"thread_start_us":599,"threads_started":1}
I20260929 16:02:36.497397 11724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b84232b57a48e2ae41ee9b84a0513c" candidate_uuid: "6bb19c0418814cc0aa842e53a10d3209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3877e25dd7574a54a1712af376dd9f5a" is_pre_election: true
I20260929 16:02:36.497395 11859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b84232b57a48e2ae41ee9b84a0513c" candidate_uuid: "6bb19c0418814cc0aa842e53a10d3209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432b8f586d7743d685b0c3095f64e1e9" is_pre_election: true
I20260929 16:02:36.497882 11724 raft_consensus.cc:2454] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb19c0418814cc0aa842e53a10d3209 in term 0.
I20260929 16:02:36.497891 11859 raft_consensus.cc:2454] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bb19c0418814cc0aa842e53a10d3209 in term 0.
I20260929 16:02:36.498530 11522 leader_election.cc:302] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3877e25dd7574a54a1712af376dd9f5a, 6bb19c0418814cc0aa842e53a10d3209; no voters:
I20260929 16:02:36.498942 11931 raft_consensus.cc:2781] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:36.499095 11931 raft_consensus.cc:482] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:36.499203 11931 raft_consensus.cc:3037] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:36.501392 11931 raft_consensus.cc:504] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } }
I20260929 16:02:36.501999 11931 leader_election.cc:288] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [CANDIDATE]: Term 1 election: Requested vote from peers 432b8f586d7743d685b0c3095f64e1e9 (127.23.222.67:39729), 3877e25dd7574a54a1712af376dd9f5a (127.23.222.66:35745)
I20260929 16:02:36.502398 11859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b84232b57a48e2ae41ee9b84a0513c" candidate_uuid: "6bb19c0418814cc0aa842e53a10d3209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "432b8f586d7743d685b0c3095f64e1e9"
I20260929 16:02:36.502444 11724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b84232b57a48e2ae41ee9b84a0513c" candidate_uuid: "6bb19c0418814cc0aa842e53a10d3209" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3877e25dd7574a54a1712af376dd9f5a"
I20260929 16:02:36.502600 11859 raft_consensus.cc:3037] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:36.502647 11724 raft_consensus.cc:3037] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [term 0 FOLLOWER]: Advancing to term 1
W20260929 16:02:36.504683 11770 tablet.cc:2369] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:36.504998 11724 raft_consensus.cc:2454] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb19c0418814cc0aa842e53a10d3209 in term 1.
I20260929 16:02:36.505120 11859 raft_consensus.cc:2454] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bb19c0418814cc0aa842e53a10d3209 in term 1.
I20260929 16:02:36.505427 11522 leader_election.cc:302] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3877e25dd7574a54a1712af376dd9f5a, 6bb19c0418814cc0aa842e53a10d3209; no voters:
I20260929 16:02:36.505719 11931 raft_consensus.cc:2781] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:36.506137 11931 raft_consensus.cc:686] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 1 LEADER]: Becoming Leader. State: Replica: 6bb19c0418814cc0aa842e53a10d3209, State: Running, Role: LEADER
I20260929 16:02:36.506584 11931 consensus_queue.cc:237] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } }
I20260929 16:02:36.511602 11442 catalog_manager.cc:5495] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bb19c0418814cc0aa842e53a10d3209 (127.23.222.65). New cstate: current_term: 1 leader_uuid: "6bb19c0418814cc0aa842e53a10d3209" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bb19c0418814cc0aa842e53a10d3209" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 40021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 } health_report { overall_health: UNKNOWN } } }
W20260929 16:02:36.525965 11906 tablet.cc:2369] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:36.529548 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.523659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.23.222.65:40021)
0929 16:02:36.523877 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.523883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.523903 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:02:36.524118 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.524123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.524226 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.524415 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:36.524786 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.525484 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.527384 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.527809 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.527950 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.528469 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:36.528986 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:36.529145 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.529206 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.529288 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:02:36.537958 11859 raft_consensus.cc:1261] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [term 1 FOLLOWER]: Refusing update from remote peer 6bb19c0418814cc0aa842e53a10d3209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:36.537958 11724 raft_consensus.cc:1261] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [term 1 FOLLOWER]: Refusing update from remote peer 6bb19c0418814cc0aa842e53a10d3209: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:36.538962 11935 consensus_queue.cc:1035] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3877e25dd7574a54a1712af376dd9f5a" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 35745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:36.539368 11931 consensus_queue.cc:1035] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "432b8f586d7743d685b0c3095f64e1e9" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 39729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:36.548944 11938 mvcc.cc:205] Tried to move back new op lower bound from 7334698010768752640 to 7334698010654625792. Current Snapshot: MvccSnapshot[applied={T|T < 7334698010768752640}]
I20260929 16:02:36.550789 11939 mvcc.cc:205] Tried to move back new op lower bound from 7334698010768752640 to 7334698010654625792. Current Snapshot: MvccSnapshot[applied={T|T < 7334698010768752640}]
I20260929 16:02:36.561319 11941 mvcc.cc:205] Tried to move back new op lower bound from 7334698010768752640 to 7334698010654625792. Current Snapshot: MvccSnapshot[applied={T|T < 7334698010768752640}]
I20260929 16:02:36.567148 11442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:36.567512 11442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:36.571664 11442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3894784c8fc24a2f8f53037839f71d55: Sending DeleteTablet for 3 replicas of tablet d3b84232b57a48e2ae41ee9b84a0513c
I20260929 16:02:36.573138 11839 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b84232b57a48e2ae41ee9b84a0513c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:36 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:57416
I20260929 16:02:36.573974 11569 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b84232b57a48e2ae41ee9b84a0513c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:36 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:41968
I20260929 16:02:36.575155 11704 tablet_service.cc:1514] Processing DeleteTablet for tablet d3b84232b57a48e2ae41ee9b84a0513c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:36 UTC) from {username='kudu', principal='kudu/127.23.222.126@KRBTEST.COM'} at 127.0.0.1:35088
I20260929 16:02:36.576458 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 11505
I20260929 16:02:36.582945 11949 tablet_replica.cc:330] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: stopping tablet replica
I20260929 16:02:36.583460 11949 raft_consensus.cc:2229] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:36.584131 11949 raft_consensus.cc:2258] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:36.588703 11949 ts_tablet_manager.cc:1905] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:36.592650 11950 tablet_replica.cc:330] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: stopping tablet replica
I20260929 16:02:36.593156 11950 raft_consensus.cc:2229] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:36.593746 11950 raft_consensus.cc:2258] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:36.594966 11948 tablet_replica.cc:330] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: stopping tablet replica
I20260929 16:02:36.600497 11950 ts_tablet_manager.cc:1905] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:36.600514 11948 raft_consensus.cc:2229] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:36.601058 11948 raft_consensus.cc:2258] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:36.603040 11948 ts_tablet_manager.cc:1905] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:36.603515 11949 ts_tablet_manager.cc:1918] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:36.603721 11949 log.cc:1196] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-0/wal/wals/d3b84232b57a48e2ae41ee9b84a0513c
I20260929 16:02:36.604319 11949 ts_tablet_manager.cc:1939] T d3b84232b57a48e2ae41ee9b84a0513c P 6bb19c0418814cc0aa842e53a10d3209: Deleting consensus metadata
I20260929 16:02:36.606329 11430 catalog_manager.cc:4841] TS 6bb19c0418814cc0aa842e53a10d3209 (127.23.222.65:40021): tablet d3b84232b57a48e2ae41ee9b84a0513c (table test-table [id=bcbfce4e0615498eb3d1e91a9a676af8]) successfully deleted
I20260929 16:02:36.608278 11950 ts_tablet_manager.cc:1918] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:36.608489 11950 log.cc:1196] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/d3b84232b57a48e2ae41ee9b84a0513c
I20260929 16:02:36.609064 11950 ts_tablet_manager.cc:1939] T d3b84232b57a48e2ae41ee9b84a0513c P 3877e25dd7574a54a1712af376dd9f5a: Deleting consensus metadata
Sep 29 16:02:36 dist-test-slave-cq88 krb5kdc[11398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.65@KRBTEST.COM
I20260929 16:02:36.611594 11948 ts_tablet_manager.cc:1918] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:36.611810 11948 log.cc:1196] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/d3b84232b57a48e2ae41ee9b84a0513c
I20260929 16:02:36.611728 11428 catalog_manager.cc:4841] TS 3877e25dd7574a54a1712af376dd9f5a (127.23.222.66:35745): tablet d3b84232b57a48e2ae41ee9b84a0513c (table test-table [id=bcbfce4e0615498eb3d1e91a9a676af8]) successfully deleted
I20260929 16:02:36.612411 11948 ts_tablet_manager.cc:1939] T d3b84232b57a48e2ae41ee9b84a0513c P 432b8f586d7743d685b0c3095f64e1e9: Deleting consensus metadata
I20260929 16:02:36.614297 11427 catalog_manager.cc:4841] TS 432b8f586d7743d685b0c3095f64e1e9 (127.23.222.67:39729): tablet d3b84232b57a48e2ae41ee9b84a0513c (table test-table [id=bcbfce4e0615498eb3d1e91a9a676af8]) successfully deleted
I20260929 16:02:36.620916 11921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.578230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.23.222.65:40021)
0929 16:02:36.578473 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.578482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.588842 (+ 10360us) server_negotiation.cc:407] Connection header received
0929 16:02:36.589521 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.589531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.589651 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.589757 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:36.594230 (+ 4473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.595505 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.597690 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.598568 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.598687 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.612765 (+ 14078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:36.612788 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:36.612811 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:36.612875 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:36.617280 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.618520 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.618530 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.618534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.618739 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.619241 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.619268 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.619273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.619745 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:36.620032 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.620389 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.620599 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 16:02:36.621809 11629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:36.721449 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 11505
I20260929 16:02:36.740173 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 11640
Sep 29 16:02:36 dist-test-slave-cq88 krb5kdc[11398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.66@KRBTEST.COM
I20260929 16:02:36.758414 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.741117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.23.222.66:35745)
0929 16:02:36.741334 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.741339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.741358 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:02:36.741414 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.741418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.741481 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.741571 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:36.742937 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.743823 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.745318 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.745958 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.746114 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.751192 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:36.751208 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:36.751222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:36.751335 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:36.754802 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.755897 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.755906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.755909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.756066 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.756520 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.756527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.756537 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.756993 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:36.757332 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.757681 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.757944 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 16:02:36.759262 11764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:36.851182 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 11640
I20260929 16:02:36.869391 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 11775
Sep 29 16:02:36 dist-test-slave-cq88 krb5kdc[11398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.67@KRBTEST.COM
I20260929 16:02:36.886125 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:36.870227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.23.222.67:39729)
0929 16:02:36.870431 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:36.870435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:36.870455 (+ 20us) server_negotiation.cc:407] Connection header received
0929 16:02:36.870627 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:36.870632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:36.870705 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:36.870821 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:36.872274 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.873072 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.874549 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:36.875012 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:36.875197 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:36.880015 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:36.880033 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:36.880050 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:36.880088 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:36.882935 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.883895 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.883903 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.883906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.884052 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:36.884490 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:36.884494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:36.884496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:36.884878 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:36.885198 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:36.885551 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 16:02:36.885794 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 16:02:36.886900 11900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:36.980585 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 11775
I20260929 16:02:36.997892 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 11411
I20260929 16:02:36.999208 11474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:37.113552 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 11411
2026-09-29T16:02:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4430 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T16:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:37Z Disabled control of system clock
I20260929 16:02:37.192978 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38271
--webserver_interface=127.23.222.126
--webserver_port=0
--builtin_ntp_servers=127.23.222.84:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:38271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:37.534312 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:37.534639 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:37.534768 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:37.542518 11960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:37.542642 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:37.542708 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:37.542798 11960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:37.542893 11960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:37.552645 11960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:38271
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:38271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:37.554076 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:37.556058 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:37.568966 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:37.569003 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:37.569041 11967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:37.569900 11960 server_base.cc:1034] running on GCE node
I20260929 16:02:37.570745 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:37.572345 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:37.573575 11960 hybrid_clock.cc:648] HybridClock initialized: now 1790697757573516 us; error 44 us; skew 500 ppm
I20260929 16:02:37.574087 11960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:37.580734 11960 webserver.cc:460] Webserver started at http://127.23.222.126:40013/ using document root <none> and password file <none>
I20260929 16:02:37.581514 11960 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:37.581640 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:37.582002 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:37.585180 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "6aba36ad562b4e538e176843f4f4b159"
format_stamp: "Formatted at 2026-09-29 16:02:37 on dist-test-slave-cq88"
server_key: "10d9b23544b7813f2993c8bd2cae6bc6"
server_key_iv: "8bbb681fcc947998ac6e0cb64dac6a50"
server_key_version: "encryptionkey@0"
I20260929 16:02:37.586058 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "6aba36ad562b4e538e176843f4f4b159"
format_stamp: "Formatted at 2026-09-29 16:02:37 on dist-test-slave-cq88"
server_key: "10d9b23544b7813f2993c8bd2cae6bc6"
server_key_iv: "8bbb681fcc947998ac6e0cb64dac6a50"
server_key_version: "encryptionkey@0"
I20260929 16:02:37.591594 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:02:37.595438 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:37.596936 11960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:37.597148 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "6aba36ad562b4e538e176843f4f4b159"
format_stamp: "Formatted at 2026-09-29 16:02:37 on dist-test-slave-cq88"
server_key: "10d9b23544b7813f2993c8bd2cae6bc6"
server_key_iv: "8bbb681fcc947998ac6e0cb64dac6a50"
server_key_version: "encryptionkey@0"
I20260929 16:02:37.597390 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:37.629010 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:37.630003 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:37.630343 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:37.650403 11960 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:38271
I20260929 16:02:37.650449 12027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:38271 every 8 connection(s)
I20260929 16:02:37.651918 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:37.655718 12028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:37.659134 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 11960
I20260929 16:02:37.659373 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:37.659940 24441 external_mini_cluster.cc:1428] Setting key 3af3981f6e9dab1503b9e297068441ec
I20260929 16:02:37.669214 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159: Bootstrap starting.
I20260929 16:02:37.672561 12028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:37.673837 12028 log.cc:824] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:37.674538 12031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:37.662052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39430 (local address 127.23.222.126:38271)
0929 16:02:37.663112 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:37.663123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:37.663166 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:02:37.665118 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:37.665141 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:37.665664 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:37.666100 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:37.666726 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:37.668388 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:37.669842 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:37.670562 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:37.670813 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:37.671972 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:37.671981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:37.671995 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:37.672531 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:37.672653 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:37.672675 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:37.672827 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:37.673052 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 16:02:37.673413 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":380,"threads_started":1}
I20260929 16:02:37.677033 12028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159: No bootstrap required, opened a new log
I20260929 16:02:37.684970 12028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba36ad562b4e538e176843f4f4b159" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38271 } }
I20260929 16:02:37.685396 12028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:37.685537 12028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aba36ad562b4e538e176843f4f4b159, State: Initialized, Role: FOLLOWER
I20260929 16:02:37.686113 12028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba36ad562b4e538e176843f4f4b159" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38271 } }
I20260929 16:02:37.686364 12028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:37.686527 12028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:37.686700 12028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:37.689327 12028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba36ad562b4e538e176843f4f4b159" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38271 } }
I20260929 16:02:37.689939 12028 leader_election.cc:302] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aba36ad562b4e538e176843f4f4b159; no voters:
I20260929 16:02:37.690701 12028 leader_election.cc:288] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:37.690862 12033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:37.691692 12033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [term 1 LEADER]: Becoming Leader. State: Replica: 6aba36ad562b4e538e176843f4f4b159, State: Running, Role: LEADER
I20260929 16:02:37.692241 12033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba36ad562b4e538e176843f4f4b159" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38271 } }
I20260929 16:02:37.692987 12028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:37.695428 12035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aba36ad562b4e538e176843f4f4b159. Latest consensus state: current_term: 1 leader_uuid: "6aba36ad562b4e538e176843f4f4b159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba36ad562b4e538e176843f4f4b159" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38271 } } }
I20260929 16:02:37.695396 12034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aba36ad562b4e538e176843f4f4b159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aba36ad562b4e538e176843f4f4b159" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 38271 } } }
I20260929 16:02:37.696099 12034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:37.696072 12035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:37.698663 12041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:37.704496 12041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:37.714452 12041 catalog_manager.cc:1347] Generated new cluster ID: b4856253df3349e3af6e178c2d91b9fa
I20260929 16:02:37.714591 12041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:37.738181 12041 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:37.739631 12041 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:37.746531 12041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159: Generated new TSK 0
I20260929 16:02:37.747682 12041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:37.764914 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.65:0
--local_ip_for_outbound_sockets=127.23.222.65
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38271
--builtin_ntp_servers=127.23.222.84:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 16:02:38.109781 12052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:38.110078 12052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:38.110224 12052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:38.117767 12052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:38.117894 12052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:38.118072 12052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.65
I20260929 16:02:38.128365 12052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.222.65
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.65
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:38.129760 12052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:38.131613 12052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:38.144826 12057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:38.144788 12058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:38.146378 12060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:38.146778 12052 server_base.cc:1034] running on GCE node
I20260929 16:02:38.147531 12052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:38.148545 12052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:38.149791 12052 hybrid_clock.cc:648] HybridClock initialized: now 1790697758149731 us; error 55 us; skew 500 ppm
I20260929 16:02:38.150151 12052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:38.152802 12052 webserver.cc:460] Webserver started at http://127.23.222.65:35297/ using document root <none> and password file <none>
I20260929 16:02:38.153642 12052 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:38.153803 12052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:38.154141 12052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:38.157305 12052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/data/instance:
uuid: "8d8f3c19f7e745f186317abb831bb69b"
format_stamp: "Formatted at 2026-09-29 16:02:38 on dist-test-slave-cq88"
server_key: "fc790f9a604594179f417b95edbb444d"
server_key_iv: "52ca36f25ce20e48d28a89569cbfb668"
server_key_version: "encryptionkey@0"
I20260929 16:02:38.158198 12052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance:
uuid: "8d8f3c19f7e745f186317abb831bb69b"
format_stamp: "Formatted at 2026-09-29 16:02:38 on dist-test-slave-cq88"
server_key: "fc790f9a604594179f417b95edbb444d"
server_key_iv: "52ca36f25ce20e48d28a89569cbfb668"
server_key_version: "encryptionkey@0"
I20260929 16:02:38.163938 12052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 16:02:38.168319 12067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:38.170069 12052 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 16:02:38.170315 12052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/wal
uuid: "8d8f3c19f7e745f186317abb831bb69b"
format_stamp: "Formatted at 2026-09-29 16:02:38 on dist-test-slave-cq88"
server_key: "fc790f9a604594179f417b95edbb444d"
server_key_iv: "52ca36f25ce20e48d28a89569cbfb668"
server_key_version: "encryptionkey@0"
I20260929 16:02:38.170593 12052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:38.186517 12052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:38.187561 12052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:38.187904 12052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:38.189023 12052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:38.190670 12052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:38.190800 12052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:38.190941 12052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:38.191030 12052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:38.226094 12052 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.65:45613
I20260929 16:02:38.226132 12180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.65:45613 every 8 connection(s)
I20260929 16:02:38.227607 12052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/data/info.pb
I20260929 16:02:38.231011 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 12052
I20260929 16:02:38.231204 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-0/wal/instance
I20260929 16:02:38.231868 24441 external_mini_cluster.cc:1428] Setting key d65325b04a6fbe3db56b51bfc7916e67
I20260929 16:02:38.234836 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.66:0
--local_ip_for_outbound_sockets=127.23.222.66
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38271
--builtin_ntp_servers=127.23.222.84:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:38.241963 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:38.230498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:54411 (local address 127.23.222.126:38271)
0929 16:02:38.231010 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:38.231015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:38.232386 (+ 1371us) server_negotiation.cc:407] Connection header received
0929 16:02:38.234586 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:38.234591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:38.234689 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:38.234830 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:38.236103 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:38.236892 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:38.238920 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:38.239420 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:38.239562 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:38.241037 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:38.241044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:38.241048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:38.241078 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:38.241120 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:38.241132 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:38.241233 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:38.241477 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 16:02:38.241586 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":274,"threads_started":1}
I20260929 16:02:38.243108 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:38.231073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38271 (local address 127.23.222.65:54411)
0929 16:02:38.232174 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:38.232214 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:38.234275 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:38.235095 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:38.235104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:38.235213 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:38.235910 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:38.235930 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:38.237057 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:38.237070 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:38.238755 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:38.238770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:38.239670 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:38.239677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:38.239926 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:38.240689 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:38.240692 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:38.240742 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:38.240830 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:38.241233 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:38.241247 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:38.241258 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:02:38.241474 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 16:02:38.241912 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":316,"threads_started":1}
I20260929 16:02:38.245077 12181 heartbeater.cc:347] Connected to a master server at 127.23.222.126:38271
I20260929 16:02:38.245446 12181 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:38.246536 12181 heartbeater.cc:511] Master 127.23.222.126:38271 requested a full tablet report, sending...
I20260929 16:02:38.248967 11993 ts_manager.cc:194] Registered new tserver with Master: 8d8f3c19f7e745f186317abb831bb69b (127.23.222.65:45613)
I20260929 16:02:38.250542 11993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.65:54411
W20260929 16:02:38.582809 12185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:38.583125 12185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:38.583288 12185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:38.590958 12185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:38.591080 12185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:38.591315 12185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.66
I20260929 16:02:38.601989 12185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.222.66
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.66
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:38.603490 12185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:38.605398 12185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:38.619058 12190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:38.620313 12191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:38.620344 12193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:38.620820 12185 server_base.cc:1034] running on GCE node
I20260929 16:02:38.621428 12185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:38.622462 12185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:38.623672 12185 hybrid_clock.cc:648] HybridClock initialized: now 1790697758623627 us; error 30 us; skew 500 ppm
I20260929 16:02:38.623984 12185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:38.626400 12185 webserver.cc:460] Webserver started at http://127.23.222.66:39313/ using document root <none> and password file <none>
I20260929 16:02:38.627108 12185 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:38.627199 12185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:38.627547 12185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:38.630491 12185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/data/instance:
uuid: "4004f37a81c24aca9369fa2d69184579"
format_stamp: "Formatted at 2026-09-29 16:02:38 on dist-test-slave-cq88"
server_key: "b7836146c9ec693a9f8b2095f52d0654"
server_key_iv: "31475775b1be6494b15920205056e41c"
server_key_version: "encryptionkey@0"
I20260929 16:02:38.631326 12185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance:
uuid: "4004f37a81c24aca9369fa2d69184579"
format_stamp: "Formatted at 2026-09-29 16:02:38 on dist-test-slave-cq88"
server_key: "b7836146c9ec693a9f8b2095f52d0654"
server_key_iv: "31475775b1be6494b15920205056e41c"
server_key_version: "encryptionkey@0"
I20260929 16:02:38.636868 12185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:02:38.640509 12199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:38.641996 12185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:02:38.642191 12185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/wal
uuid: "4004f37a81c24aca9369fa2d69184579"
format_stamp: "Formatted at 2026-09-29 16:02:38 on dist-test-slave-cq88"
server_key: "b7836146c9ec693a9f8b2095f52d0654"
server_key_iv: "31475775b1be6494b15920205056e41c"
server_key_version: "encryptionkey@0"
I20260929 16:02:38.642428 12185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:38.662406 12185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:38.663368 12185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:38.663739 12185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:38.664788 12185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:38.666465 12185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:38.666605 12185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:38.666744 12185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:38.666838 12185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:38.701527 12185 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.66:36417
I20260929 16:02:38.701555 12312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.66:36417 every 8 connection(s)
I20260929 16:02:38.702952 12185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/data/info.pb
I20260929 16:02:38.704850 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 12185
I20260929 16:02:38.705057 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/wal/instance
I20260929 16:02:38.705574 24441 external_mini_cluster.cc:1428] Setting key 9da94b6ce3c64310b5a10abfdf072c7e
I20260929 16:02:38.709067 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.222.67:0
--local_ip_for_outbound_sockets=127.23.222.67
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38271
--builtin_ntp_servers=127.23.222.84:32949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 16:02:38.719035 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:38.705931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.66:51051 (local address 127.23.222.126:38271)
0929 16:02:38.706164 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:38.706172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:38.707815 (+ 1643us) server_negotiation.cc:407] Connection header received
0929 16:02:38.710701 (+ 2886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:38.710706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:38.710772 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:38.710876 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:38.712435 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:38.713143 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:38.715761 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:38.716151 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:38.716252 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:38.717959 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:38.717964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:38.717967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:38.718020 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:38.718071 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:38.718089 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:38.718193 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:38.718537 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 16:02:38.718656 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:02:38.719969 12315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:38.706592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38271 (local address 127.23.222.66:51051)
0929 16:02:38.707654 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:38.707693 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:38.710353 (+ 2660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:38.711166 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:38.711181 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:38.711353 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:38.712220 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:38.712249 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:38.713301 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:38.713309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:38.715599 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:38.715617 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:38.716407 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:38.716417 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:38.716744 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:38.717558 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:38.717563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:38.717623 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:38.717742 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:38.718224 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:38.718234 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:38.718240 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 16:02:38.718442 (+ 202us) client_negotiation.cc:240] Negotiation successful
0929 16:02:38.718767 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":322,"threads_started":1}
I20260929 16:02:38.721558 12313 heartbeater.cc:347] Connected to a master server at 127.23.222.126:38271
I20260929 16:02:38.721887 12313 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:38.722697 12313 heartbeater.cc:511] Master 127.23.222.126:38271 requested a full tablet report, sending...
I20260929 16:02:38.724243 11993 ts_manager.cc:194] Registered new tserver with Master: 4004f37a81c24aca9369fa2d69184579 (127.23.222.66:36417)
I20260929 16:02:38.725133 11993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.66:51051
W20260929 16:02:39.055315 12316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:39.055648 12316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:39.055814 12316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:39.063447 12316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:39.063568 12316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:39.063678 12316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.222.67
I20260929 16:02:39.073055 12316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:32949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.222.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.222.67
--webserver_port=0
--tserver_master_addrs=127.23.222.126:38271
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.222.67
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:39.074332 12316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:39.076332 12316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:39.090220 12321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:39.090170 12322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:39.090209 12324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:39.091040 12316 server_base.cc:1034] running on GCE node
I20260929 16:02:39.091750 12316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:39.092833 12316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:39.094076 12316 hybrid_clock.cc:648] HybridClock initialized: now 1790697759094000 us; error 73 us; skew 500 ppm
I20260929 16:02:39.094468 12316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 16:02:39.096946 12316 webserver.cc:460] Webserver started at http://127.23.222.67:39389/ using document root <none> and password file <none>
I20260929 16:02:39.097715 12316 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:39.097818 12316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:39.098173 12316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:39.101481 12316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/data/instance:
uuid: "f619336321224bfeb71a61bc01d928fd"
format_stamp: "Formatted at 2026-09-29 16:02:39 on dist-test-slave-cq88"
server_key: "ad46b34887358e06047e5684f1393c82"
server_key_iv: "60f7629cdea26a213839cd1505aebed8"
server_key_version: "encryptionkey@0"
I20260929 16:02:39.102412 12316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance:
uuid: "f619336321224bfeb71a61bc01d928fd"
format_stamp: "Formatted at 2026-09-29 16:02:39 on dist-test-slave-cq88"
server_key: "ad46b34887358e06047e5684f1393c82"
server_key_iv: "60f7629cdea26a213839cd1505aebed8"
server_key_version: "encryptionkey@0"
I20260929 16:02:39.107753 12316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:02:39.111301 12330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:39.112751 12316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:39.112929 12316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/wal
uuid: "f619336321224bfeb71a61bc01d928fd"
format_stamp: "Formatted at 2026-09-29 16:02:39 on dist-test-slave-cq88"
server_key: "ad46b34887358e06047e5684f1393c82"
server_key_iv: "60f7629cdea26a213839cd1505aebed8"
server_key_version: "encryptionkey@0"
I20260929 16:02:39.113157 12316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:39.129804 12316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:39.130735 12316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:39.131069 12316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:39.132160 12316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 16:02:39.133713 12316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 16:02:39.133803 12316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:39.133970 12316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 16:02:39.134066 12316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:39.168473 12316 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.67:37321
I20260929 16:02:39.168520 12443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.67:37321 every 8 connection(s)
I20260929 16:02:39.169831 12316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/data/info.pb
I20260929 16:02:39.179155 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 12316
I20260929 16:02:39.179406 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/wal/instance
I20260929 16:02:39.179957 24441 external_mini_cluster.cc:1428] Setting key 876c9962ad1fa42c2e547caedb1316a8
I20260929 16:02:39.182965 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.172523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.67:41085 (local address 127.23.222.126:38271)
0929 16:02:39.172739 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.172746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.174117 (+ 1371us) server_negotiation.cc:407] Connection header received
0929 16:02:39.176026 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.176030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.176098 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.176184 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:39.177229 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.178126 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.179960 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.180558 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.180704 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.182119 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:39.182124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:39.182128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:39.182145 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:39.182177 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:39.182189 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:39.182287 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.182517 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.182619 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:02:39.184089 12446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.173091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:38271 (local address 127.23.222.67:41085)
0929 16:02:39.173969 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.174007 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:39.175740 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:39.176385 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:39.176393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:39.176499 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:02:39.177049 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.177065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.178259 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.178263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.179805 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.179817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.180808 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.180815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.181055 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.181775 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 16:02:39.181778 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:02:39.181816 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 16:02:39.181903 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:02:39.182287 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:02:39.182300 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:02:39.182311 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 16:02:39.182506 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 16:02:39.182920 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":266,"threads_started":1}
I20260929 16:02:39.185619 12444 heartbeater.cc:347] Connected to a master server at 127.23.222.126:38271
I20260929 16:02:39.185961 12444 heartbeater.cc:464] Registering TS with master...
I20260929 16:02:39.186739 12444 heartbeater.cc:511] Master 127.23.222.126:38271 requested a full tablet report, sending...
I20260929 16:02:39.188292 11993 ts_manager.cc:194] Registered new tserver with Master: f619336321224bfeb71a61bc01d928fd (127.23.222.67:37321)
I20260929 16:02:39.189100 11993 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.222.67:41085
I20260929 16:02:39.194993 24441 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 16:02:39.205153 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.199142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.23.222.126:38271)
0929 16:02:39.199371 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.199376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.199879 (+ 503us) server_negotiation.cc:407] Connection header received
0929 16:02:39.200024 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.200028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.200099 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.200174 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:39.200618 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.201419 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.202741 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.203171 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.203372 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.204441 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:39.204446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:39.204449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:39.204467 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:39.204510 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:39.204523 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:39.204674 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.204827 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.204914 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:02:39.209267 11993 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 16:02:39.216210 11993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 16:02:39.244354 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.229695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.23.222.66:36417)
0929 16:02:39.233321 (+ 3626us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.233337 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.234524 (+ 1187us) server_negotiation.cc:407] Connection header received
0929 16:02:39.235015 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.235025 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.235322 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.235480 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:39.236708 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.238157 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.241196 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.242604 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.242716 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.242982 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.243793 (+ 811us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.243981 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3454,"thread_start_us":3328,"threads_started":1}
I20260929 16:02:39.244371 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.229679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.23.222.67:37321)
0929 16:02:39.233875 (+ 4196us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.233888 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.234522 (+ 634us) server_negotiation.cc:407] Connection header received
0929 16:02:39.234984 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.234993 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.235302 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.235480 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:39.236449 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.237877 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.241217 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.242636 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.242765 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.243037 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.243825 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.244010 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3985,"thread_start_us":3811,"threads_started":1}
I20260929 16:02:39.244395 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.231363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:36417 (local address 127.0.0.1:51918)
0929 16:02:39.234388 (+ 3025us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.234409 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:39.234713 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:39.235814 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:39.235821 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:39.235857 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:39.236469 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.236481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.238325 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.238332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.240812 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.240832 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.243203 (+ 2371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.243212 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.243480 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.243609 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 16:02:39.243747 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 16:02:39.243962 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2881,"mutex_wait_us":108,"spinlock_wait_cycles":7936,"thread_start_us":2537,"threads_started":1}
I20260929 16:02:39.244413 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.235357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.23.222.65:45613)
0929 16:02:39.236157 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.236165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.236832 (+ 667us) server_negotiation.cc:407] Connection header received
0929 16:02:39.237044 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.237053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.237301 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.237493 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:39.238082 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.239574 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.241675 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.243110 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.243271 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.243547 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.243902 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.244056 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":479,"threads_started":1}
I20260929 16:02:39.245119 12459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.229154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:37321 (local address 127.0.0.1:50496)
0929 16:02:39.234361 (+ 5207us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.234402 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:39.234713 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:39.235640 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:39.235648 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:39.235677 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:39.236153 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.236168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.238039 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.238044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.240838 (+ 2794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.240847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.243150 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.243155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.243497 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.243609 (+ 112us) client_negotiation.cc:769] Sending connection context
0929 16:02:39.244638 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0929 16:02:39.244766 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4932,"mutex_wait_us":80,"spinlock_wait_cycles":4864,"thread_start_us":4924,"threads_started":1}
I20260929 16:02:39.245420 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.234818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45613 (local address 127.0.0.1:46258)
0929 16:02:39.236692 (+ 1874us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.236715 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:39.236840 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:39.237507 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:39.237512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:39.237538 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:39.237864 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.237885 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.239718 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.239723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.241471 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.241489 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.243337 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.243345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.243607 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.243633 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 16:02:39.243733 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 16:02:39.244976 (+ 1243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":1090,"thread_start_us":1576,"threads_started":1}
I20260929 16:02:39.248800 12115 tablet_service.cc:1467] Processing CreateTablet for tablet f861395409354aaeadc143fdacb3bbef (DEFAULT_TABLE table=test-table [id=b7e2e18736e24ac39481ea7b0b28664c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:39.248855 12378 tablet_service.cc:1467] Processing CreateTablet for tablet f861395409354aaeadc143fdacb3bbef (DEFAULT_TABLE table=test-table [id=b7e2e18736e24ac39481ea7b0b28664c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:39.248850 12247 tablet_service.cc:1467] Processing CreateTablet for tablet f861395409354aaeadc143fdacb3bbef (DEFAULT_TABLE table=test-table [id=b7e2e18736e24ac39481ea7b0b28664c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 16:02:39.250101 12115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f861395409354aaeadc143fdacb3bbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:39.250167 12378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f861395409354aaeadc143fdacb3bbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:39.250167 12247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f861395409354aaeadc143fdacb3bbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:39.253682 12181 heartbeater.cc:503] Master 127.23.222.126:38271 was elected leader, sending a full tablet report...
I20260929 16:02:39.262784 12462 tablet_bootstrap.cc:492] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b: Bootstrap starting.
I20260929 16:02:39.266371 12462 tablet_bootstrap.cc:654] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:39.267720 12462 log.cc:824] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:39.268038 12464 tablet_bootstrap.cc:492] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: Bootstrap starting.
I20260929 16:02:39.270118 12463 tablet_bootstrap.cc:492] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: Bootstrap starting.
I20260929 16:02:39.271817 12462 tablet_bootstrap.cc:492] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b: No bootstrap required, opened a new log
I20260929 16:02:39.272034 12464 tablet_bootstrap.cc:654] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:39.272099 12462 ts_tablet_manager.cc:1397] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 16:02:39.273253 12464 log.cc:824] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:39.273751 12463 tablet_bootstrap.cc:654] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:39.275152 12463 log.cc:824] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:39.277240 12464 tablet_bootstrap.cc:492] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: No bootstrap required, opened a new log
I20260929 16:02:39.277554 12464 ts_tablet_manager.cc:1397] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 16:02:39.278268 12463 tablet_bootstrap.cc:492] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: No bootstrap required, opened a new log
I20260929 16:02:39.278523 12463 ts_tablet_manager.cc:1397] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 16:02:39.280602 12462 raft_consensus.cc:348] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } }
I20260929 16:02:39.280998 12462 raft_consensus.cc:374] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:39.281111 12462 raft_consensus.cc:729] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8f3c19f7e745f186317abb831bb69b, State: Initialized, Role: FOLLOWER
I20260929 16:02:39.281740 12462 consensus_queue.cc:260] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } }
I20260929 16:02:39.283416 12462 ts_tablet_manager.cc:1428] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 16:02:39.288738 12463 raft_consensus.cc:348] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } }
I20260929 16:02:39.289137 12463 raft_consensus.cc:374] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:39.289243 12463 raft_consensus.cc:729] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f619336321224bfeb71a61bc01d928fd, State: Initialized, Role: FOLLOWER
I20260929 16:02:39.289384 12464 raft_consensus.cc:348] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } }
I20260929 16:02:39.289875 12464 raft_consensus.cc:374] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:39.289980 12464 raft_consensus.cc:729] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4004f37a81c24aca9369fa2d69184579, State: Initialized, Role: FOLLOWER
I20260929 16:02:39.290002 12463 consensus_queue.cc:260] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } }
I20260929 16:02:39.290644 12464 consensus_queue.cc:260] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } }
I20260929 16:02:39.291180 12444 heartbeater.cc:503] Master 127.23.222.126:38271 was elected leader, sending a full tablet report...
I20260929 16:02:39.291774 12463 ts_tablet_manager.cc:1428] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20260929 16:02:39.291808 12313 heartbeater.cc:503] Master 127.23.222.126:38271 was elected leader, sending a full tablet report...
I20260929 16:02:39.293820 12464 ts_tablet_manager.cc:1428] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260929 16:02:39.421607 12445 tablet.cc:2369] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:39.455184 12314 tablet.cc:2369] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 16:02:39.480257 12182 tablet.cc:2369] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:02:39.529630 12468 raft_consensus.cc:482] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 16:02:39.529982 12468 raft_consensus.cc:504] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } }
I20260929 16:02:39.531709 12468 leader_election.cc:288] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f619336321224bfeb71a61bc01d928fd (127.23.222.67:37321), 4004f37a81c24aca9369fa2d69184579 (127.23.222.66:36417)
I20260929 16:02:39.538354 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.532236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:60903 (local address 127.23.222.67:37321)
0929 16:02:39.532524 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.532533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.533168 (+ 635us) server_negotiation.cc:407] Connection header received
0929 16:02:39.533413 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.533417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.533484 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.533582 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:39.534100 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.534813 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.536686 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.537438 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.537634 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.537708 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.537981 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.538107 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260929 16:02:39.538465 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.532143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.67:37321 (local address 127.23.222.65:60903)
0929 16:02:39.533037 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.533063 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:39.533257 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:39.533602 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:39.533607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:39.533631 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:39.533950 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.533959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.534944 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.534952 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.536489 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.536502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.537635 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.537641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.537804 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.537829 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 16:02:39.537937 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 16:02:39.538082 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":3,"thread_start_us":531,"threads_started":1}
I20260929 16:02:39.539244 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.532433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.222.65:41993 (local address 127.23.222.66:36417)
0929 16:02:39.532797 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.532807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.533517 (+ 710us) server_negotiation.cc:407] Connection header received
0929 16:02:39.533742 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.533753 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.533843 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.533958 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 16:02:39.534553 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.535340 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.537237 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.538074 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.538256 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.538336 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.538792 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.538954 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260929 16:02:39.539746 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.532328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.66:36417 (local address 127.23.222.65:41993)
0929 16:02:39.533399 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.533415 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:02:39.533589 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:02:39.533986 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:02:39.533991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:02:39.534018 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 16:02:39.534375 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.534384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.535504 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.535510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.537037 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:02:39.537051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.538307 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.538314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:02:39.538549 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.538602 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 16:02:39.539317 (+ 715us) client_negotiation.cc:240] Negotiation successful
0929 16:02:39.539494 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":697,"threads_started":1}
I20260929 16:02:39.539716 12398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f861395409354aaeadc143fdacb3bbef" candidate_uuid: "8d8f3c19f7e745f186317abb831bb69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f619336321224bfeb71a61bc01d928fd" is_pre_election: true
I20260929 16:02:39.540411 12398 raft_consensus.cc:2454] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d8f3c19f7e745f186317abb831bb69b in term 0.
I20260929 16:02:39.540863 12267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f861395409354aaeadc143fdacb3bbef" candidate_uuid: "8d8f3c19f7e745f186317abb831bb69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4004f37a81c24aca9369fa2d69184579" is_pre_election: true
I20260929 16:02:39.541160 12071 leader_election.cc:302] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d8f3c19f7e745f186317abb831bb69b, f619336321224bfeb71a61bc01d928fd; no voters:
I20260929 16:02:39.541406 12267 raft_consensus.cc:2454] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d8f3c19f7e745f186317abb831bb69b in term 0.
I20260929 16:02:39.541657 12468 raft_consensus.cc:2781] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 16:02:39.541802 12468 raft_consensus.cc:482] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 16:02:39.541955 12468 raft_consensus.cc:3037] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:39.544219 12468 raft_consensus.cc:504] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } }
I20260929 16:02:39.544842 12468 leader_election.cc:288] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [CANDIDATE]: Term 1 election: Requested vote from peers f619336321224bfeb71a61bc01d928fd (127.23.222.67:37321), 4004f37a81c24aca9369fa2d69184579 (127.23.222.66:36417)
I20260929 16:02:39.545217 12398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f861395409354aaeadc143fdacb3bbef" candidate_uuid: "8d8f3c19f7e745f186317abb831bb69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f619336321224bfeb71a61bc01d928fd"
I20260929 16:02:39.545333 12267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f861395409354aaeadc143fdacb3bbef" candidate_uuid: "8d8f3c19f7e745f186317abb831bb69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4004f37a81c24aca9369fa2d69184579"
I20260929 16:02:39.545503 12398 raft_consensus.cc:3037] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:39.545578 12267 raft_consensus.cc:3037] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:39.548007 12398 raft_consensus.cc:2454] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d8f3c19f7e745f186317abb831bb69b in term 1.
I20260929 16:02:39.548026 12267 raft_consensus.cc:2454] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d8f3c19f7e745f186317abb831bb69b in term 1.
I20260929 16:02:39.548466 12069 leader_election.cc:302] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4004f37a81c24aca9369fa2d69184579, 8d8f3c19f7e745f186317abb831bb69b; no voters:
I20260929 16:02:39.548801 12468 raft_consensus.cc:2781] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:39.549299 12468 raft_consensus.cc:686] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 1 LEADER]: Becoming Leader. State: Replica: 8d8f3c19f7e745f186317abb831bb69b, State: Running, Role: LEADER
I20260929 16:02:39.549784 12468 consensus_queue.cc:237] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } }
I20260929 16:02:39.554296 11992 catalog_manager.cc:5495] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d8f3c19f7e745f186317abb831bb69b (127.23.222.65). New cstate: current_term: 1 leader_uuid: "8d8f3c19f7e745f186317abb831bb69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d8f3c19f7e745f186317abb831bb69b" member_type: VOTER last_known_addr { host: "127.23.222.65" port: 45613 } health_report { overall_health: HEALTHY } } }
I20260929 16:02:39.613698 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.607793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.23.222.65:45613)
0929 16:02:39.608065 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.608071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.608222 (+ 151us) server_negotiation.cc:407] Connection header received
0929 16:02:39.608504 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.608508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.608602 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.608769 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:02:39.609159 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.609866 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.611372 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.611829 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.611972 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.612521 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:02:39.613108 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:02:39.613278 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.613335 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.613432 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:02:39.622440 12267 raft_consensus.cc:1261] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [term 1 FOLLOWER]: Refusing update from remote peer 8d8f3c19f7e745f186317abb831bb69b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:39.622440 12398 raft_consensus.cc:1261] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [term 1 FOLLOWER]: Refusing update from remote peer 8d8f3c19f7e745f186317abb831bb69b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:02:39.623549 12473 consensus_queue.cc:1035] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f619336321224bfeb71a61bc01d928fd" member_type: VOTER last_known_addr { host: "127.23.222.67" port: 37321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:39.623966 12468 consensus_queue.cc:1035] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4004f37a81c24aca9369fa2d69184579" member_type: VOTER last_known_addr { host: "127.23.222.66" port: 36417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:02:39.651146 11992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:39.651546 11992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 16:02:39.655355 11992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6aba36ad562b4e538e176843f4f4b159: Sending DeleteTablet for 3 replicas of tablet f861395409354aaeadc143fdacb3bbef
I20260929 16:02:39.656610 12247 tablet_service.cc:1514] Processing DeleteTablet for tablet f861395409354aaeadc143fdacb3bbef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:39 UTC) from {username='slave'} at 127.0.0.1:51918
I20260929 16:02:39.656859 12378 tablet_service.cc:1514] Processing DeleteTablet for tablet f861395409354aaeadc143fdacb3bbef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:39 UTC) from {username='slave'} at 127.0.0.1:50496
I20260929 16:02:39.659479 12115 tablet_service.cc:1514] Processing DeleteTablet for tablet f861395409354aaeadc143fdacb3bbef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 16:02:39 UTC) from {username='slave'} at 127.0.0.1:46258
I20260929 16:02:39.663789 12488 tablet_replica.cc:330] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b: stopping tablet replica
I20260929 16:02:39.664408 12487 tablet_replica.cc:330] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: stopping tablet replica
I20260929 16:02:39.665031 12488 raft_consensus.cc:2229] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 1 LEADER]: Raft consensus shutting down.
I20260929 16:02:39.665729 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12052
I20260929 16:02:39.668767 12488 raft_consensus.cc:2258] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:39.668985 12487 raft_consensus.cc:2229] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:39.669632 12487 raft_consensus.cc:2258] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:39.674511 12487 ts_tablet_manager.cc:1905] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:39.685453 12486 tablet_replica.cc:330] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: stopping tablet replica
I20260929 16:02:39.685907 12486 raft_consensus.cc:2229] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 16:02:39.686424 12486 raft_consensus.cc:2258] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 16:02:39.689424 12486 ts_tablet_manager.cc:1905] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:39.672392 12488 ts_tablet_manager.cc:1905] T f861395409354aaeadc143fdacb3bbef P 8d8f3c19f7e745f186317abb831bb69b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 16:02:39.691208 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.667744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.23.222.65:45613)
0929 16:02:39.667996 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.668006 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.678960 (+ 10954us) server_negotiation.cc:407] Connection header received
0929 16:02:39.679602 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.679611 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.679779 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.679881 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:39.680854 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.682501 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.686884 (+ 4383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.687489 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.687611 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.689255 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:39.689271 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:39.689300 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:39.690154 (+ 854us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:39.690229 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:39.690259 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:39.690363 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.690644 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.690780 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 16:02:39.693259 12176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:39.691214 12487 ts_tablet_manager.cc:1918] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:39.694468 12487 log.cc:1196] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-2/wal/wals/f861395409354aaeadc143fdacb3bbef
I20260929 16:02:39.695071 12487 ts_tablet_manager.cc:1939] T f861395409354aaeadc143fdacb3bbef P f619336321224bfeb71a61bc01d928fd: Deleting consensus metadata
I20260929 16:02:39.697244 11979 catalog_manager.cc:4841] TS f619336321224bfeb71a61bc01d928fd (127.23.222.67:37321): tablet f861395409354aaeadc143fdacb3bbef (table test-table [id=b7e2e18736e24ac39481ea7b0b28664c]) successfully deleted
I20260929 16:02:39.699509 12486 ts_tablet_manager.cc:1918] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 16:02:39.699729 12486 log.cc:1196] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: Deleting WAL directory at /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790697431711191-24441-0/minicluster-data/ts-1/wal/wals/f861395409354aaeadc143fdacb3bbef
I20260929 16:02:39.700198 12486 ts_tablet_manager.cc:1939] T f861395409354aaeadc143fdacb3bbef P 4004f37a81c24aca9369fa2d69184579: Deleting consensus metadata
I20260929 16:02:39.702284 11977 catalog_manager.cc:4841] TS 4004f37a81c24aca9369fa2d69184579 (127.23.222.66:36417): tablet f861395409354aaeadc143fdacb3bbef (table test-table [id=b7e2e18736e24ac39481ea7b0b28664c]) successfully deleted
I20260929 16:02:39.808508 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 12052
W20260929 16:02:39.829193 11977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.222.65:45613 (error 108)
I20260929 16:02:39.830385 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12185
I20260929 16:02:39.831195 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.830415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45613 (local address 127.0.0.1:46294)
0929 16:02:39.830890 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.830994 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":220,"threads_started":1}
W20260929 16:02:39.831646 11977 catalog_manager.cc:4584] TS 8d8f3c19f7e745f186317abb831bb69b (127.23.222.65:45613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f861395409354aaeadc143fdacb3bbef: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45613: connect: Connection refused (error 111)
I20260929 16:02:39.837747 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.831108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.23.222.66:36417)
0929 16:02:39.831329 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.831336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.831382 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:02:39.831581 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.831585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.831666 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.831767 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:39.832255 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.833119 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.834621 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.835065 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.835173 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.836281 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:39.836289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:39.836304 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:39.836946 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:39.836999 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:39.837020 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:39.837119 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.837237 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.837336 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:02:39.838501 12308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:39.885249 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.884763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45613 (local address 127.0.0.1:46304)
0929 16:02:39.885015 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.885103 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 16:02:39.886471 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.886036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45613 (local address 127.0.0.1:46316)
0929 16:02:39.886215 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.886283 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 16:02:39.930496 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 12185
I20260929 16:02:39.947307 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.946760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45613 (local address 127.0.0.1:46324)
0929 16:02:39.947002 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.947094 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 16:02:39.948236 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12316
I20260929 16:02:39.948609 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.948186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45613 (local address 127.0.0.1:46334)
0929 16:02:39.948342 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:39.948425 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 16:02:39.955977 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:39.949017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.23.222.67:37321)
0929 16:02:39.949199 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:39.949204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:39.949222 (+ 18us) server_negotiation.cc:407] Connection header received
0929 16:02:39.949314 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:39.949317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:39.949403 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:39.949492 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:39.949860 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.950999 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.953090 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:39.953569 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:39.953670 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:39.954601 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:39.954609 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:39.954625 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:39.955229 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:39.955413 (+ 184us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:39.955440 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:39.955540 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:39.955642 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 16:02:39.955728 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 16:02:39.956709 12439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:40.029536 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:40.028986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45613 (local address 127.0.0.1:46342)
0929 16:02:40.029259 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:40.029373 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 16:02:40.030738 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:40.030281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.65:45613 (local address 127.0.0.1:46356)
0929 16:02:40.030513 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 16:02:40.030589 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.65:45613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260929 16:02:40.047406 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 12316
I20260929 16:02:40.064564 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 11960
I20260929 16:02:40.065773 12023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:40.179996 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 11960
2026-09-29T16:02:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3070 ms)
[----------] 6 tests from EncryptionPolicyTest (23349 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:40 dist-test-slave-cq88 krb5kdc[12498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:40 dist-test-slave-cq88 krb5kdc[12498](info): set up 2 sockets
Sep 29 16:02:40 dist-test-slave-cq88 krb5kdc[12498](info): commencing operation
krb5kdc: starting...
W20260929 16:02:41.330830 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:41 dist-test-slave-cq88 krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:41.603621 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:41.943331 12511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:41.943717 12511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:41.943890 12511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:41.951288 12511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:41.951467 12511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:41.951586 12511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:41.951699 12511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:41.951786 12511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:41.963572 12511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:41.965008 12511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:41.966368 12511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:41.978994 12519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:41.979031 12516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:41.978963 12517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:41.980186 12511 server_base.cc:1034] running on GCE node
I20260929 16:02:41.981001 12511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:41.982484 12511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:41.983693 12511 hybrid_clock.cc:648] HybridClock initialized: now 1790697761983631 us; error 59 us; skew 500 ppm
I20260929 16:02:41.984035 12511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:41 dist-test-slave-cq88 krb5kdc[12498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:41.987317 12511 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:41.988917 12511 webserver.cc:460] Webserver started at http://127.0.0.1:44717/ using document root <none> and password file <none>
I20260929 16:02:41.989593 12511 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:41.989698 12511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:41.989946 12511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:41.993157 12511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "ea5c61be093741e8b0cf09dbf2f59e08"
format_stamp: "Formatted at 2026-09-29 16:02:41 on dist-test-slave-cq88"
server_key: "04dcbfdfe94da684f3097cb12d97bc26"
server_key_iv: "5901b86aec6fdd4649a31140d82336fb"
server_key_version: "encryptionkey@0"
I20260929 16:02:41.993956 12511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "ea5c61be093741e8b0cf09dbf2f59e08"
format_stamp: "Formatted at 2026-09-29 16:02:41 on dist-test-slave-cq88"
server_key: "04dcbfdfe94da684f3097cb12d97bc26"
server_key_iv: "5901b86aec6fdd4649a31140d82336fb"
server_key_version: "encryptionkey@0"
I20260929 16:02:41.999378 12511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:02:42.002964 12526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:42.004463 12511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:02:42.004627 12511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "ea5c61be093741e8b0cf09dbf2f59e08"
format_stamp: "Formatted at 2026-09-29 16:02:41 on dist-test-slave-cq88"
server_key: "04dcbfdfe94da684f3097cb12d97bc26"
server_key_iv: "5901b86aec6fdd4649a31140d82336fb"
server_key_version: "encryptionkey@0"
I20260929 16:02:42.004806 12511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:42.020517 12511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:42.025273 12511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:42.025571 12511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:42.046213 12511 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40379
I20260929 16:02:42.046235 12578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40379 every 8 connection(s)
I20260929 16:02:42.047761 12511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:42.050416 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 12511
I20260929 16:02:42.050652 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:42.051548 24441 external_mini_cluster.cc:1428] Setting key 2ef695f5c3678caed923569b07bd960c
I20260929 16:02:42.052454 12579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:42.065505 12579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08: Bootstrap starting.
Sep 29 16:02:42 dist-test-slave-cq88 krb5kdc[12498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:02:42.069800 12579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:42.071272 12579 log.cc:824] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:42.075201 12579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08: No bootstrap required, opened a new log
I20260929 16:02:42.076495 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:42.053781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51907 (local address 127.0.0.1:40379)
0929 16:02:42.054627 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:42.054639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:42.054681 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:02:42.056404 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:42.056428 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:42.056862 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:42.057304 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:42.058823 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:42.060246 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:42.061584 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:42.062285 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:42.062544 (+ 259us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:42.066778 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:42.066790 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:42.066810 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:42.066866 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:42.070986 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:42.072094 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:42.072104 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:42.072111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:42.072281 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:42.072888 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:42.072895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:42.072897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:42.073800 (+ 903us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:42.074176 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:42.074709 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 16:02:42.075067 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":372,"threads_started":1}
I20260929 16:02:42.080755 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:02:42.084270 12579 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5c61be093741e8b0cf09dbf2f59e08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260929 16:02:42.084714 12579 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:42.084823 12579 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5c61be093741e8b0cf09dbf2f59e08, State: Initialized, Role: FOLLOWER
I20260929 16:02:42.085523 12579 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5c61be093741e8b0cf09dbf2f59e08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260929 16:02:42.085835 12579 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:42.086014 12579 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:42.086182 12579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:42.088580 12579 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5c61be093741e8b0cf09dbf2f59e08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260929 16:02:42.089090 12579 leader_election.cc:302] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea5c61be093741e8b0cf09dbf2f59e08; no voters:
I20260929 16:02:42.089771 12579 leader_election.cc:288] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:42.089888 12589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:42.092521 12579 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:42.095937 12589 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [term 1 LEADER]: Becoming Leader. State: Replica: ea5c61be093741e8b0cf09dbf2f59e08, State: Running, Role: LEADER
I20260929 16:02:42.096884 12589 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5c61be093741e8b0cf09dbf2f59e08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } }
I20260929 16:02:42.101310 12591 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea5c61be093741e8b0cf09dbf2f59e08. Latest consensus state: current_term: 1 leader_uuid: "ea5c61be093741e8b0cf09dbf2f59e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5c61be093741e8b0cf09dbf2f59e08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } } }
I20260929 16:02:42.101898 12591 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:42.101738 12590 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea5c61be093741e8b0cf09dbf2f59e08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea5c61be093741e8b0cf09dbf2f59e08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40379 } } }
I20260929 16:02:42.102501 12590 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:42.103938 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:42.085535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45311 (local address 127.0.0.1:40379)
0929 16:02:42.085698 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:42.085702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:42.086371 (+ 669us) server_negotiation.cc:407] Connection header received
0929 16:02:42.086587 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:42.086592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:42.086665 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:42.086773 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:42.088257 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:42.089115 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:42.090757 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:42.091281 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:42.091433 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:42.094140 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:42.094145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:42.094149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:42.094183 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:42.098594 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:42.099745 (+ 1151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:42.099749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:42.099752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:42.100132 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:42.100842 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:42.100851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:42.100856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:42.101355 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:42.101675 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:42.103076 (+ 1401us) server_negotiation.cc:299] Negotiation successful
0929 16:02:42.103391 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 16:02:42.112954 12596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:42.118263 12596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:42.128206 12596 catalog_manager.cc:1347] Generated new cluster ID: ae5ae0b1c79b4ce0930745501168d2cf
I20260929 16:02:42.128335 12596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:42.139333 12596 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:42.141144 12596 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:42.155476 12596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea5c61be093741e8b0cf09dbf2f59e08: Generated new TSK 0
I20260929 16:02:42.156383 12596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:42.166642 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12511
I20260929 16:02:42.168296 12574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:42.263309 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 12511
W20260929 16:02:42.281754 12496 connection.cc:535] client connection to 127.0.0.1:40379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40379): Connection reset by peer (error 104)
2026-09-29T16:02:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:42 dist-test-slave-cq88 krb5kdc[12611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:42 dist-test-slave-cq88 krb5kdc[12611](info): set up 2 sockets
Sep 29 16:02:42 dist-test-slave-cq88 krb5kdc[12611](info): commencing operation
krb5kdc: starting...
W20260929 16:02:43.370648 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:43 dist-test-slave-cq88 krb5kdc[12611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:43.639911 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38613
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:43.982314 12624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:43.982672 12624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:43.982846 12624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:43.990387 12624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:43.990556 12624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:43.990682 12624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:43.990795 12624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:43.990875 12624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:44.001423 12624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38613
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:44.002846 12624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:44.004276 12624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:44.017256 12630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:44.017407 12629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:44.017324 12632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:44.018841 12624 server_base.cc:1034] running on GCE node
I20260929 16:02:44.019869 12624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:44.021440 12624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:44.022672 12624 hybrid_clock.cc:648] HybridClock initialized: now 1790697764022605 us; error 74 us; skew 500 ppm
I20260929 16:02:44.023030 12624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:44 dist-test-slave-cq88 krb5kdc[12611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:44.033913 12624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:44.035818 12624 webserver.cc:460] Webserver started at http://127.0.0.1:45235/ using document root <none> and password file <none>
I20260929 16:02:44.036585 12624 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:44.036707 12624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:44.036981 12624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:44.040370 12624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "9505f386e9844f64876e1552ac1ae937"
format_stamp: "Formatted at 2026-09-29 16:02:44 on dist-test-slave-cq88"
server_key: "8694622108c9a3b5c0008edbf64a87d6"
server_key_iv: "7f9d591ecf5c7848e0bd6bf8829437e7"
server_key_version: "encryptionkey@0"
I20260929 16:02:44.041167 12624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "9505f386e9844f64876e1552ac1ae937"
format_stamp: "Formatted at 2026-09-29 16:02:44 on dist-test-slave-cq88"
server_key: "8694622108c9a3b5c0008edbf64a87d6"
server_key_iv: "7f9d591ecf5c7848e0bd6bf8829437e7"
server_key_version: "encryptionkey@0"
I20260929 16:02:44.047084 12624 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 16:02:44.051329 12639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:44.052968 12624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:44.053181 12624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "9505f386e9844f64876e1552ac1ae937"
format_stamp: "Formatted at 2026-09-29 16:02:44 on dist-test-slave-cq88"
server_key: "8694622108c9a3b5c0008edbf64a87d6"
server_key_iv: "7f9d591ecf5c7848e0bd6bf8829437e7"
server_key_version: "encryptionkey@0"
I20260929 16:02:44.053421 12624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:44.076539 12624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:44.081535 12624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:44.081863 12624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:44.102190 12624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38613
I20260929 16:02:44.102237 12690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38613 every 8 connection(s)
I20260929 16:02:44.103677 12624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:44.107051 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 12624
I20260929 16:02:44.107224 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:44.107620 12691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:44.107667 24441 external_mini_cluster.cc:1428] Setting key acbe480b22e3899fea2aa4f1dc60adfc
I20260929 16:02:44.122175 12691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937: Bootstrap starting.
I20260929 16:02:44.122452 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.109392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60841 (local address 127.0.0.1:38613)
0929 16:02:44.110204 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.110216 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.110257 (+ 41us) server_negotiation.cc:407] Connection header received
0929 16:02:44.112374 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.112398 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.112813 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.113152 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:44.113797 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.115176 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.117161 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.118163 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.118322 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.119808 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:44.119817 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:44.119836 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:44.120467 (+ 631us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:44.120549 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:44.120582 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:44.120710 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.121122 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.121336 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":378,"threads_started":1}
I20260929 16:02:44.126157 12691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:44.126819 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:02:44.127478 12691 log.cc:824] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:44.131191 12691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937: No bootstrap required, opened a new log
I20260929 16:02:44.139938 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:44.132164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59073 (local address 127.0.0.1:38613)
0929 16:02:44.132389 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:44.132398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:44.132825 (+ 427us) server_negotiation.cc:407] Connection header received
0929 16:02:44.133095 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:44.133099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:44.133171 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:44.133289 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:44.133886 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.134777 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.136571 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:44.137108 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:44.137289 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:44.138828 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:44.138834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:44.138837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:44.138860 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:44.138906 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:44.138928 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:44.139106 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:44.139537 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 16:02:44.139605 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:02:44.141089 12691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505f386e9844f64876e1552ac1ae937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38613 } }
I20260929 16:02:44.141644 12691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:44.141772 12691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9505f386e9844f64876e1552ac1ae937, State: Initialized, Role: FOLLOWER
I20260929 16:02:44.142436 12691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505f386e9844f64876e1552ac1ae937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38613 } }
I20260929 16:02:44.142680 12691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:44.142777 12691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:44.142915 12691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:44.145318 12691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505f386e9844f64876e1552ac1ae937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38613 } }
I20260929 16:02:44.145850 12691 leader_election.cc:302] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9505f386e9844f64876e1552ac1ae937; no voters:
I20260929 16:02:44.146531 12691 leader_election.cc:288] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:44.146703 12702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:44.148056 12702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [term 1 LEADER]: Becoming Leader. State: Replica: 9505f386e9844f64876e1552ac1ae937, State: Running, Role: LEADER
I20260929 16:02:44.148811 12702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505f386e9844f64876e1552ac1ae937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38613 } }
I20260929 16:02:44.149137 12691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:44.152143 12704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9505f386e9844f64876e1552ac1ae937. Latest consensus state: current_term: 1 leader_uuid: "9505f386e9844f64876e1552ac1ae937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505f386e9844f64876e1552ac1ae937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38613 } } }
I20260929 16:02:44.152684 12704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:44.153263 12703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9505f386e9844f64876e1552ac1ae937" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505f386e9844f64876e1552ac1ae937" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38613 } } }
I20260929 16:02:44.153620 12703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:44.158063 12710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:44.163918 12710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:44.174232 12710 catalog_manager.cc:1347] Generated new cluster ID: a47997e1b9944a8fa3d7c31d587c7b76
I20260929 16:02:44.174369 12710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:44.202726 12710 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:44.204308 12710 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:44.211797 12710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9505f386e9844f64876e1552ac1ae937: Generated new TSK 0
I20260929 16:02:44.212837 12710 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:44.218974 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12624
W20260929 16:02:44.220312 12686 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60841
W20260929 16:02:44.221306 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12624: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:02:44.221511 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 12624
W20260929 16:02:44.240293 12609 connection.cc:535] client connection to 127.0.0.1:38613 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38613): Connection reset by peer (error 104)
2026-09-29T16:02:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:44 dist-test-slave-cq88 krb5kdc[12723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:44 dist-test-slave-cq88 krb5kdc[12723](info): set up 2 sockets
Sep 29 16:02:44 dist-test-slave-cq88 krb5kdc[12723](info): commencing operation
krb5kdc: starting...
W20260929 16:02:45.342697 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:45 dist-test-slave-cq88 krb5kdc[12723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:45.610184 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:45.953948 12737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:45.954360 12737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:45.954553 12737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:45.962349 12737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:45.962566 12737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:45.962678 12737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:45.962791 12737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:45.962873 12737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:45.973518 12737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:45.975848 12737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:45.977674 12737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:45.990900 12743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:45.990948 12745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:45.991470 12742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:45.991680 12737 server_base.cc:1034] running on GCE node
I20260929 16:02:45.992772 12737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:45.994313 12737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:45.995538 12737 hybrid_clock.cc:648] HybridClock initialized: now 1790697765995468 us; error 63 us; skew 500 ppm
I20260929 16:02:45.995981 12737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:45 dist-test-slave-cq88 krb5kdc[12723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:45.999527 12737 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:46.001384 12737 webserver.cc:460] Webserver started at http://127.0.0.1:42397/ using document root <none> and password file <none>
I20260929 16:02:46.002178 12737 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:46.002323 12737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:46.002676 12737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:46.006141 12737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "ecd333194def4060a07a742f72dbe246"
format_stamp: "Formatted at 2026-09-29 16:02:45 on dist-test-slave-cq88"
server_key: "e5d6701859652ecf4a1495b220cee063"
server_key_iv: "cf473e25a413791a2e30a823b6417edf"
server_key_version: "encryptionkey@0"
I20260929 16:02:46.007036 12737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "ecd333194def4060a07a742f72dbe246"
format_stamp: "Formatted at 2026-09-29 16:02:45 on dist-test-slave-cq88"
server_key: "e5d6701859652ecf4a1495b220cee063"
server_key_iv: "cf473e25a413791a2e30a823b6417edf"
server_key_version: "encryptionkey@0"
I20260929 16:02:46.012892 12737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 16:02:46.016932 12752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:46.018460 12737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:02:46.018659 12737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "ecd333194def4060a07a742f72dbe246"
format_stamp: "Formatted at 2026-09-29 16:02:45 on dist-test-slave-cq88"
server_key: "e5d6701859652ecf4a1495b220cee063"
server_key_iv: "cf473e25a413791a2e30a823b6417edf"
server_key_version: "encryptionkey@0"
I20260929 16:02:46.018883 12737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:46.025307 12737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:46.030398 12737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:46.030773 12737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:46.050838 12737 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36173
I20260929 16:02:46.050890 12804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36173 every 8 connection(s)
I20260929 16:02:46.052465 12737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:46.056417 12805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:46.056751 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 12737
I20260929 16:02:46.056929 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:46.057353 24441 external_mini_cluster.cc:1428] Setting key cffc5a32734f04e5603ebf980ae4ca49
I20260929 16:02:46.066201 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:46.059231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39955 (local address 127.0.0.1:36173)
0929 16:02:46.060289 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:46.060305 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:46.060358 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:02:46.062054 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:46.062094 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:46.062592 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:46.063104 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:46.064180 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:46.064189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:46.064207 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:46.064224 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:46.064314 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:46.064347 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:46.064578 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:46.064728 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 16:02:46.064937 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":385,"threads_started":1}
I20260929 16:02:46.070006 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:02:46.071748 12805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246: Bootstrap starting.
I20260929 16:02:46.075088 12805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:46.076205 12805 log.cc:824] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:46.076994 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:46.074478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56571 (local address 127.0.0.1:36173)
0929 16:02:46.074684 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:46.074691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:46.074975 (+ 284us) server_negotiation.cc:407] Connection header received
0929 16:02:46.075193 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:46.075215 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:46.075301 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:46.075423 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:46.076320 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:46.076325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:46.076331 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:46.076351 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:46.076395 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:46.076429 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:46.076546 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:46.076643 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 16:02:46.076709 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 16:02:46.079075 12805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246: No bootstrap required, opened a new log
I20260929 16:02:46.086557 12805 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd333194def4060a07a742f72dbe246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } }
I20260929 16:02:46.086963 12805 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:46.087118 12805 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd333194def4060a07a742f72dbe246, State: Initialized, Role: FOLLOWER
I20260929 16:02:46.087839 12805 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd333194def4060a07a742f72dbe246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } }
I20260929 16:02:46.088091 12805 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:46.088234 12805 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:46.088420 12805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:46.090746 12805 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd333194def4060a07a742f72dbe246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } }
I20260929 16:02:46.091351 12805 leader_election.cc:302] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecd333194def4060a07a742f72dbe246; no voters:
I20260929 16:02:46.091984 12805 leader_election.cc:288] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:46.092167 12815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:46.092897 12815 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [term 1 LEADER]: Becoming Leader. State: Replica: ecd333194def4060a07a742f72dbe246, State: Running, Role: LEADER
I20260929 16:02:46.093533 12815 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd333194def4060a07a742f72dbe246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } }
I20260929 16:02:46.094492 12805 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:46.096591 12816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecd333194def4060a07a742f72dbe246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd333194def4060a07a742f72dbe246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } } }
I20260929 16:02:46.096724 12817 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecd333194def4060a07a742f72dbe246. Latest consensus state: current_term: 1 leader_uuid: "ecd333194def4060a07a742f72dbe246" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd333194def4060a07a742f72dbe246" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36173 } } }
I20260929 16:02:46.097011 12816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:46.097571 12817 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:46.098101 12823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:46.103802 12823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:46.113519 12823 catalog_manager.cc:1347] Generated new cluster ID: 522eac03a9c842c7abcfaa62f8ccc6a1
I20260929 16:02:46.113686 12823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:46.126313 12823 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:46.126778 12823 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:46.135442 12823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecd333194def4060a07a742f72dbe246: Generated new TSK 0
I20260929 16:02:46.136360 12823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:46.149899 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12737
W20260929 16:02:46.151672 12800 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39955
W20260929 16:02:46.152397 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12737: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:02:46.152566 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 12737
2026-09-29T16:02:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1930 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:46 dist-test-slave-cq88 krb5kdc[12836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:46 dist-test-slave-cq88 krb5kdc[12836](info): set up 2 sockets
Sep 29 16:02:46 dist-test-slave-cq88 krb5kdc[12836](info): commencing operation
krb5kdc: starting...
W20260929 16:02:47.278733 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:47 dist-test-slave-cq88 krb5kdc[12836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:47.551044 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:47.891471 12849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:47.891785 12849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:47.891976 12849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:47.899525 12849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:47.899650 12849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:47.899756 12849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:47.899868 12849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:47.899948 12849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:47.910131 12849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:47.911561 12849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:47.912909 12849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:47.925196 12854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:47.925226 12857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:47.925195 12855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:47.926064 12849 server_base.cc:1034] running on GCE node
I20260929 16:02:47.926898 12849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:47.928427 12849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:47.929663 12849 hybrid_clock.cc:648] HybridClock initialized: now 1790697767929618 us; error 62 us; skew 500 ppm
I20260929 16:02:47.930022 12849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:47 dist-test-slave-cq88 krb5kdc[12836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:47.933236 12849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:47.934798 12849 webserver.cc:460] Webserver started at http://127.0.0.1:41269/ using document root <none> and password file <none>
I20260929 16:02:47.935550 12849 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:47.935653 12849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:47.935971 12849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:47.939170 12849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "86f9816ab08a4b92889703f8ab48c3b9"
format_stamp: "Formatted at 2026-09-29 16:02:47 on dist-test-slave-cq88"
server_key: "9b265604636b6847e32970ea0228f196"
server_key_iv: "982f942e0ae0d4f7cfdd785b8b9a1483"
server_key_version: "encryptionkey@0"
I20260929 16:02:47.940009 12849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "86f9816ab08a4b92889703f8ab48c3b9"
format_stamp: "Formatted at 2026-09-29 16:02:47 on dist-test-slave-cq88"
server_key: "9b265604636b6847e32970ea0228f196"
server_key_iv: "982f942e0ae0d4f7cfdd785b8b9a1483"
server_key_version: "encryptionkey@0"
I20260929 16:02:47.945284 12849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:02:47.948952 12864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:47.950309 12849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:02:47.950520 12849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "86f9816ab08a4b92889703f8ab48c3b9"
format_stamp: "Formatted at 2026-09-29 16:02:47 on dist-test-slave-cq88"
server_key: "9b265604636b6847e32970ea0228f196"
server_key_iv: "982f942e0ae0d4f7cfdd785b8b9a1483"
server_key_version: "encryptionkey@0"
I20260929 16:02:47.950716 12849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:47.970085 12849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:47.974337 12849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:47.974666 12849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:47.995196 12849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45191
I20260929 16:02:47.995227 12916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45191 every 8 connection(s)
I20260929 16:02:47.996762 12849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:47.997222 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 12849
I20260929 16:02:47.997418 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:47.998073 24441 external_mini_cluster.cc:1428] Setting key b10c7c2e4941426dc9035ac02802dbbc
I20260929 16:02:48.001780 12917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 16:02:48 dist-test-slave-cq88 krb5kdc[12836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:02:48.015117 12917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9: Bootstrap starting.
I20260929 16:02:48.019992 12917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:48.021299 12917 log.cc:824] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:48.025120 12917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9: No bootstrap required, opened a new log
I20260929 16:02:48.025177 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:47.999984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48707 (local address 127.0.0.1:45191)
0929 16:02:48.000784 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:48.000799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:48.000851 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:02:48.003260 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:48.003293 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:48.003824 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:48.004274 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:48.005864 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.008063 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.009573 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.010652 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.010855 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:48.015464 (+ 4609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:48.015477 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:48.015511 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:48.015577 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:48.019995 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:48.020962 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:48.020972 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:48.020981 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:48.021226 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:48.021699 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:48.021704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:48.021709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:48.022441 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:48.022703 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:48.023293 (+ 590us) server_negotiation.cc:299] Negotiation successful
0929 16:02:48.023779 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":326,"threads_started":1}
I20260929 16:02:48.029459 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:02:48.033978 12917 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9816ab08a4b92889703f8ab48c3b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45191 } }
I20260929 16:02:48.034549 12917 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:48.034673 12917 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f9816ab08a4b92889703f8ab48c3b9, State: Initialized, Role: FOLLOWER
I20260929 16:02:48.035473 12917 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9816ab08a4b92889703f8ab48c3b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45191 } }
I20260929 16:02:48.035799 12917 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:48.035959 12917 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:48.036164 12917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:48.039350 12917 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9816ab08a4b92889703f8ab48c3b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45191 } }
I20260929 16:02:48.040066 12917 leader_election.cc:302] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f9816ab08a4b92889703f8ab48c3b9; no voters:
I20260929 16:02:48.040783 12917 leader_election.cc:288] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:48.040920 12927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:48.044701 12927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [term 1 LEADER]: Becoming Leader. State: Replica: 86f9816ab08a4b92889703f8ab48c3b9, State: Running, Role: LEADER
I20260929 16:02:48.045732 12927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9816ab08a4b92889703f8ab48c3b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45191 } }
I20260929 16:02:48.047178 12917 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:48.051076 12929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f9816ab08a4b92889703f8ab48c3b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9816ab08a4b92889703f8ab48c3b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45191 } } }
I20260929 16:02:48.051522 12929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:48.054657 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:48.034067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46653 (local address 127.0.0.1:45191)
0929 16:02:48.034329 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:48.034336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:48.034738 (+ 402us) server_negotiation.cc:407] Connection header received
0929 16:02:48.034956 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:48.034960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:48.035075 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:48.035202 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:48.037216 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.038031 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.039854 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:48.040398 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:48.040551 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:48.044621 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:48.044629 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:48.044634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:48.044702 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:48.048325 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:48.049545 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:48.049555 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:48.049562 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:48.049832 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:48.050492 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:48.050517 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:48.050523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:48.051093 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:48.051487 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:48.053054 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0929 16:02:48.053401 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 16:02:48.054637 12928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f9816ab08a4b92889703f8ab48c3b9. Latest consensus state: current_term: 1 leader_uuid: "86f9816ab08a4b92889703f8ab48c3b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f9816ab08a4b92889703f8ab48c3b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45191 } } }
I20260929 16:02:48.055927 12928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:48.056773 12934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:48.061865 12934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:48.071376 12934 catalog_manager.cc:1347] Generated new cluster ID: f71f2142dbea4b228139331a018cb521
I20260929 16:02:48.071527 12934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:48.088655 12934 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:48.090056 12934 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:48.102551 12934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86f9816ab08a4b92889703f8ab48c3b9: Generated new TSK 0
I20260929 16:02:48.103554 12934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:48.120630 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12849
I20260929 16:02:48.121922 12912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:48.219641 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 12849
2026-09-29T16:02:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2064 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:48 dist-test-slave-cq88 krb5kdc[12949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:48 dist-test-slave-cq88 krb5kdc[12949](info): set up 2 sockets
Sep 29 16:02:48 dist-test-slave-cq88 krb5kdc[12949](info): commencing operation
krb5kdc: starting...
W20260929 16:02:49.346599 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:49 dist-test-slave-cq88 krb5kdc[12949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:49.623937 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:49.973387 12962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:49.973788 12962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:49.973955 12962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:49.981426 12962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:49.981609 12962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:49.981724 12962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:49.981840 12962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:49.981918 12962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:49.993721 12962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:49.995153 12962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:49.996634 12962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:50.009806 12968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:50.009774 12970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:50.009774 12967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:50.011343 12962 server_base.cc:1034] running on GCE node
I20260929 16:02:50.012214 12962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:50.013751 12962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:50.014987 12962 hybrid_clock.cc:648] HybridClock initialized: now 1790697770014945 us; error 33 us; skew 500 ppm
I20260929 16:02:50.015508 12962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:50 dist-test-slave-cq88 krb5kdc[12949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:50.023973 12962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:50.025750 12962 webserver.cc:460] Webserver started at http://127.0.0.1:33063/ using document root <none> and password file <none>
I20260929 16:02:50.026499 12962 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:50.026640 12962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:50.026974 12962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:50.030229 12962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "5325bac70bbe465e9e72f3de9dbbbda2"
format_stamp: "Formatted at 2026-09-29 16:02:50 on dist-test-slave-cq88"
server_key: "3faa4e8150dd4106e0652ceffee705ee"
server_key_iv: "d3a3428cb0edaab0d72467517ead69f3"
server_key_version: "encryptionkey@0"
I20260929 16:02:50.031075 12962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "5325bac70bbe465e9e72f3de9dbbbda2"
format_stamp: "Formatted at 2026-09-29 16:02:50 on dist-test-slave-cq88"
server_key: "3faa4e8150dd4106e0652ceffee705ee"
server_key_iv: "d3a3428cb0edaab0d72467517ead69f3"
server_key_version: "encryptionkey@0"
I20260929 16:02:50.036651 12962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:02:50.040560 12977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:50.042027 12962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:50.042240 12962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "5325bac70bbe465e9e72f3de9dbbbda2"
format_stamp: "Formatted at 2026-09-29 16:02:50 on dist-test-slave-cq88"
server_key: "3faa4e8150dd4106e0652ceffee705ee"
server_key_iv: "d3a3428cb0edaab0d72467517ead69f3"
server_key_version: "encryptionkey@0"
I20260929 16:02:50.042459 12962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:50.073357 12962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:50.077860 12962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:50.078172 12962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:50.097929 12962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38279
I20260929 16:02:50.097977 13029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38279 every 8 connection(s)
I20260929 16:02:50.099460 12962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:50.101598 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 12962
I20260929 16:02:50.101785 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:50.102232 24441 external_mini_cluster.cc:1428] Setting key 158064ab7af76b2cca4f06c5d4cd2fc4
I20260929 16:02:50.103816 13030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:50.116444 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:50.104118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54367 (local address 127.0.0.1:38279)
0929 16:02:50.104913 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:50.104927 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:50.104970 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:02:50.106548 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:50.106571 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:50.106981 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:50.107465 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:50.108067 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:50.109597 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:50.111628 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:50.112355 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:50.112522 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:50.113797 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:50.113806 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:50.113821 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:50.114433 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:50.114525 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:50.114552 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:50.114647 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:50.114982 (+ 335us) server_negotiation.cc:299] Negotiation successful
0929 16:02:50.115420 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":370,"threads_started":1}
I20260929 16:02:50.117594 13030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2: Bootstrap starting.
I20260929 16:02:50.120347 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:02:50.121073 13030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:50.122324 13030 log.cc:824] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:50.125487 13030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2: No bootstrap required, opened a new log
I20260929 16:02:50.132527 13033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:50.125739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53011 (local address 127.0.0.1:38279)
0929 16:02:50.125914 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:50.125921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:50.126293 (+ 372us) server_negotiation.cc:407] Connection header received
0929 16:02:50.126510 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:50.126515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:50.126589 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:50.126702 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:50.127293 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:50.128462 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:50.129815 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:50.130484 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:50.130632 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:50.131707 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:50.131714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:50.131719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:50.131742 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:50.131817 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:50.131844 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:50.131953 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:50.132058 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:02:50.132166 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 16:02:50.135766 13030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325bac70bbe465e9e72f3de9dbbbda2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38279 } }
I20260929 16:02:50.136150 13030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:50.136224 13030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5325bac70bbe465e9e72f3de9dbbbda2, State: Initialized, Role: FOLLOWER
I20260929 16:02:50.136742 13030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325bac70bbe465e9e72f3de9dbbbda2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38279 } }
I20260929 16:02:50.136955 13030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:50.137058 13030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:50.137187 13030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:50.139945 13030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325bac70bbe465e9e72f3de9dbbbda2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38279 } }
I20260929 16:02:50.140642 13030 leader_election.cc:302] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5325bac70bbe465e9e72f3de9dbbbda2; no voters:
I20260929 16:02:50.141273 13030 leader_election.cc:288] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:50.141453 13040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:50.142112 13040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [term 1 LEADER]: Becoming Leader. State: Replica: 5325bac70bbe465e9e72f3de9dbbbda2, State: Running, Role: LEADER
I20260929 16:02:50.142724 13040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325bac70bbe465e9e72f3de9dbbbda2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38279 } }
I20260929 16:02:50.144712 13040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5325bac70bbe465e9e72f3de9dbbbda2. Latest consensus state: current_term: 1 leader_uuid: "5325bac70bbe465e9e72f3de9dbbbda2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325bac70bbe465e9e72f3de9dbbbda2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38279 } } }
I20260929 16:02:50.145085 13040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:50.145555 13042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5325bac70bbe465e9e72f3de9dbbbda2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5325bac70bbe465e9e72f3de9dbbbda2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38279 } } }
I20260929 16:02:50.145969 13042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:50.149501 13030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:50.150789 13044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:50.156569 13044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:50.166188 13044 catalog_manager.cc:1347] Generated new cluster ID: 19a7c456d7154cafafb72035058b3b8d
I20260929 16:02:50.166318 13044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:50.182798 13044 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:50.184976 13044 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:50.198208 13044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5325bac70bbe465e9e72f3de9dbbbda2: Generated new TSK 0
I20260929 16:02:50.199452 13044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:50.216766 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12962
W20260929 16:02:50.218241 13025 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54367
W20260929 16:02:50.219173 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 12962: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:02:50.219358 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 12962
2026-09-29T16:02:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:50 dist-test-slave-cq88 krb5kdc[13061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:50 dist-test-slave-cq88 krb5kdc[13061](info): set up 2 sockets
Sep 29 16:02:50 dist-test-slave-cq88 krb5kdc[13061](info): commencing operation
krb5kdc: starting...
W20260929 16:02:51.326666 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:51 dist-test-slave-cq88 krb5kdc[13061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:51.619644 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:51.962750 13074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:51.963129 13074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:51.963326 13074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:51.970829 13074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:51.971014 13074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:51.971136 13074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:51.971282 13074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:51.971374 13074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:51.981796 13074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:51.983196 13074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:51.984589 13074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:51.997238 13079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:51.997850 13082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:51.997268 13080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:51.998605 13074 server_base.cc:1034] running on GCE node
I20260929 16:02:51.999831 13074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:52.001741 13074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:52.003001 13074 hybrid_clock.cc:648] HybridClock initialized: now 1790697772002955 us; error 41 us; skew 500 ppm
I20260929 16:02:52.003494 13074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:52 dist-test-slave-cq88 krb5kdc[13061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:52.006883 13074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:52.008621 13074 webserver.cc:460] Webserver started at http://127.0.0.1:40783/ using document root <none> and password file <none>
I20260929 16:02:52.009369 13074 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:52.009524 13074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:52.009845 13074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:52.013058 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "d3f558685d384b10858ec28fa1542a10"
format_stamp: "Formatted at 2026-09-29 16:02:52 on dist-test-slave-cq88"
server_key: "a091de7b2051e169ece20e028051cc3f"
server_key_iv: "4b3236ca9cd0d80a4b32208a34803b52"
server_key_version: "encryptionkey@0"
I20260929 16:02:52.013890 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "d3f558685d384b10858ec28fa1542a10"
format_stamp: "Formatted at 2026-09-29 16:02:52 on dist-test-slave-cq88"
server_key: "a091de7b2051e169ece20e028051cc3f"
server_key_iv: "4b3236ca9cd0d80a4b32208a34803b52"
server_key_version: "encryptionkey@0"
I20260929 16:02:52.019671 13074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:02:52.023710 13089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:52.025285 13074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:52.025522 13074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "d3f558685d384b10858ec28fa1542a10"
format_stamp: "Formatted at 2026-09-29 16:02:52 on dist-test-slave-cq88"
server_key: "a091de7b2051e169ece20e028051cc3f"
server_key_iv: "4b3236ca9cd0d80a4b32208a34803b52"
server_key_version: "encryptionkey@0"
I20260929 16:02:52.025759 13074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:52.032419 13074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:52.037496 13074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:52.037865 13074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:52.058674 13074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33279
I20260929 16:02:52.058719 13140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33279 every 8 connection(s)
I20260929 16:02:52.060215 13074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:52.064517 13141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:52.066484 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 13074
I20260929 16:02:52.066707 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:52.067168 24441 external_mini_cluster.cc:1428] Setting key 8abbf4510a7bcb43c6c82428aa7be615
I20260929 16:02:52.076215 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.069276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39107 (local address 127.0.0.1:33279)
0929 16:02:52.070255 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.070268 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.070314 (+ 46us) server_negotiation.cc:407] Connection header received
0929 16:02:52.072163 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.072190 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.072589 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.072940 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:52.074245 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:52.074252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:52.074266 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:52.074284 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:52.074370 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:52.074399 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:52.074616 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.074721 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.075001 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":460,"threads_started":1}
I20260929 16:02:52.077590 13141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10: Bootstrap starting.
I20260929 16:02:52.080607 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:02:52.081431 13141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:52.082473 13141 log.cc:824] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:52.085855 13141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10: No bootstrap required, opened a new log
I20260929 16:02:52.088187 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:52.085330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40085 (local address 127.0.0.1:33279)
0929 16:02:52.085565 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:52.085572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:52.085971 (+ 399us) server_negotiation.cc:407] Connection header received
0929 16:02:52.086136 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:52.086140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:52.086191 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:52.086277 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:52.087442 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:52.087446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:52.087449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:52.087467 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:52.087514 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:52.087530 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:52.087630 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:52.087791 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 16:02:52.087848 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 16:02:52.094630 13141 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f558685d384b10858ec28fa1542a10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } }
I20260929 16:02:52.095065 13141 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:52.095175 13141 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f558685d384b10858ec28fa1542a10, State: Initialized, Role: FOLLOWER
I20260929 16:02:52.095814 13141 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f558685d384b10858ec28fa1542a10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } }
I20260929 16:02:52.096062 13141 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:52.096220 13141 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:52.096395 13141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:52.099064 13141 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f558685d384b10858ec28fa1542a10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } }
I20260929 16:02:52.099638 13141 leader_election.cc:302] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3f558685d384b10858ec28fa1542a10; no voters:
I20260929 16:02:52.100255 13141 leader_election.cc:288] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:52.100453 13151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:52.101300 13151 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [term 1 LEADER]: Becoming Leader. State: Replica: d3f558685d384b10858ec28fa1542a10, State: Running, Role: LEADER
I20260929 16:02:52.101902 13151 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f558685d384b10858ec28fa1542a10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } }
I20260929 16:02:52.102496 13141 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:52.104785 13153 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3f558685d384b10858ec28fa1542a10. Latest consensus state: current_term: 1 leader_uuid: "d3f558685d384b10858ec28fa1542a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f558685d384b10858ec28fa1542a10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } } }
I20260929 16:02:52.104908 13152 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3f558685d384b10858ec28fa1542a10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3f558685d384b10858ec28fa1542a10" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33279 } } }
I20260929 16:02:52.105242 13153 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:52.105329 13152 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:52.106251 13159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:52.113050 13159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:52.123824 13159 catalog_manager.cc:1347] Generated new cluster ID: a550ddba9e854f6bb6e9987f014ce422
I20260929 16:02:52.123957 13159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:52.152868 13159 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:52.153389 13159 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:52.161523 13159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3f558685d384b10858ec28fa1542a10: Generated new TSK 0
I20260929 16:02:52.162590 13159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:52.176172 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13074
W20260929 16:02:52.177352 13136 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39107
W20260929 16:02:52.178170 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13074: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:02:52.178342 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 13074
2026-09-29T16:02:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:52 dist-test-slave-cq88 krb5kdc[13173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:52 dist-test-slave-cq88 krb5kdc[13173](info): set up 2 sockets
Sep 29 16:02:52 dist-test-slave-cq88 krb5kdc[13173](info): commencing operation
krb5kdc: starting...
W20260929 16:02:53.295859 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:53 dist-test-slave-cq88 krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:53.571661 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:53.918947 13186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:53.919382 13186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:53.919579 13186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:53.927166 13186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:53.927402 13186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:53.927516 13186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:53.927608 13186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:53.927672 13186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:53.937877 13186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:53.939311 13186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:53.940685 13186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:53.953087 13192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:53.953161 13191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:53.953226 13194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:53.954550 13186 server_base.cc:1034] running on GCE node
I20260929 16:02:53.955642 13186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:53.957297 13186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:53.958572 13186 hybrid_clock.cc:648] HybridClock initialized: now 1790697773958531 us; error 57 us; skew 500 ppm
I20260929 16:02:53.959002 13186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:53 dist-test-slave-cq88 krb5kdc[13173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:53.962489 13186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:53.964141 13186 webserver.cc:460] Webserver started at http://127.0.0.1:33363/ using document root <none> and password file <none>
I20260929 16:02:53.964815 13186 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:53.964918 13186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:53.965179 13186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:53.968297 13186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "67f32b58dc2d436bbf7de827ca8566f3"
format_stamp: "Formatted at 2026-09-29 16:02:53 on dist-test-slave-cq88"
server_key: "acb76ca8a1c8ac9e248335c3fdb4ae25"
server_key_iv: "3b0e38bfadf62257f29f4b4916dc8170"
server_key_version: "encryptionkey@0"
I20260929 16:02:53.969041 13186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "67f32b58dc2d436bbf7de827ca8566f3"
format_stamp: "Formatted at 2026-09-29 16:02:53 on dist-test-slave-cq88"
server_key: "acb76ca8a1c8ac9e248335c3fdb4ae25"
server_key_iv: "3b0e38bfadf62257f29f4b4916dc8170"
server_key_version: "encryptionkey@0"
I20260929 16:02:53.974280 13186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 16:02:53.978207 13201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:53.979842 13186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 16:02:53.980070 13186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "67f32b58dc2d436bbf7de827ca8566f3"
format_stamp: "Formatted at 2026-09-29 16:02:53 on dist-test-slave-cq88"
server_key: "acb76ca8a1c8ac9e248335c3fdb4ae25"
server_key_iv: "3b0e38bfadf62257f29f4b4916dc8170"
server_key_version: "encryptionkey@0"
I20260929 16:02:53.980289 13186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:53.994867 13186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:53.999568 13186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:53.999917 13186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:54.021916 13186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35129
I20260929 16:02:54.021935 13252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35129 every 8 connection(s)
I20260929 16:02:54.023526 13186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:54.027557 13253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:54.028640 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 13186
I20260929 16:02:54.028833 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:54.029389 24441 external_mini_cluster.cc:1428] Setting key 869d46828be286b40ea91fe9d79e840f
I20260929 16:02:54.042646 13253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3: Bootstrap starting.
Sep 29 16:02:54 dist-test-slave-cq88 krb5kdc[13173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:02:54.046954 13253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:54.048363 13253 log.cc:824] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:54.051501 13253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3: No bootstrap required, opened a new log
I20260929 16:02:54.054737 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:54.031345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:44559 (local address 127.0.0.1:35129)
0929 16:02:54.032205 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:54.032218 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:54.032262 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:02:54.034168 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:54.034195 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:54.034680 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:54.035065 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:54.037092 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.038538 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.040534 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.041321 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.041508 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:54.046783 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:54.046793 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:54.046809 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:54.046867 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:54.050286 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:54.051149 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:54.051155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:54.051162 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:54.051403 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:54.051940 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:54.051944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:54.051946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:54.052473 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:54.052806 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:54.053366 (+ 560us) server_negotiation.cc:299] Negotiation successful
0929 16:02:54.053687 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":380,"threads_started":1}
I20260929 16:02:54.058848 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:02:54.060084 13253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f32b58dc2d436bbf7de827ca8566f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260929 16:02:54.060465 13253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:54.060576 13253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f32b58dc2d436bbf7de827ca8566f3, State: Initialized, Role: FOLLOWER
I20260929 16:02:54.061127 13253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f32b58dc2d436bbf7de827ca8566f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260929 16:02:54.061374 13253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:54.061542 13253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:54.061735 13253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:54.064620 13253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f32b58dc2d436bbf7de827ca8566f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260929 16:02:54.065332 13253 leader_election.cc:302] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67f32b58dc2d436bbf7de827ca8566f3; no voters:
I20260929 16:02:54.066015 13253 leader_election.cc:288] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:54.066126 13263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:54.067662 13263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [term 1 LEADER]: Becoming Leader. State: Replica: 67f32b58dc2d436bbf7de827ca8566f3, State: Running, Role: LEADER
I20260929 16:02:54.068259 13263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f32b58dc2d436bbf7de827ca8566f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } }
I20260929 16:02:54.068450 13253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:54.070894 13265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67f32b58dc2d436bbf7de827ca8566f3. Latest consensus state: current_term: 1 leader_uuid: "67f32b58dc2d436bbf7de827ca8566f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f32b58dc2d436bbf7de827ca8566f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } } }
I20260929 16:02:54.071295 13265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:54.071846 13264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67f32b58dc2d436bbf7de827ca8566f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67f32b58dc2d436bbf7de827ca8566f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35129 } } }
I20260929 16:02:54.072146 13264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:54.083022 13268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:54.085435 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:54.063535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:50493 (local address 127.0.0.1:35129)
0929 16:02:54.063784 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:54.063793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:54.064192 (+ 399us) server_negotiation.cc:407] Connection header received
0929 16:02:54.064445 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:54.064450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:54.064516 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:54.064609 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:54.066667 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.067900 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.069823 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:54.070830 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:54.070980 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:54.075298 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:54.075308 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:54.075316 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:02:54.075384 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:54.080127 (+ 4743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:54.081533 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:54.081542 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:54.081549 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:54.081748 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:02:54.082448 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:02:54.082456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:02:54.082461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:02:54.083028 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:54.083375 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:54.084618 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0929 16:02:54.084875 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:02:54.088950 13268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:54.098340 13268 catalog_manager.cc:1347] Generated new cluster ID: 6d603a13a0fc4133b8f8ccc9dba02ba1
I20260929 16:02:54.098515 13268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:54.130165 13268 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:54.131573 13268 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:54.139058 13268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67f32b58dc2d436bbf7de827ca8566f3: Generated new TSK 0
I20260929 16:02:54.140023 13268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:54.159298 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13186
I20260929 16:02:54.160543 13248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:02:54.259506 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 13186
W20260929 16:02:54.277426 13171 connection.cc:535] client connection to 127.0.0.1:35129 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35129): Connection reset by peer (error 104)
2026-09-29T16:02:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2079 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:54 dist-test-slave-cq88 krb5kdc[13285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:54 dist-test-slave-cq88 krb5kdc[13285](info): set up 2 sockets
Sep 29 16:02:54 dist-test-slave-cq88 krb5kdc[13285](info): commencing operation
krb5kdc: starting...
W20260929 16:02:55.394763 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:55 dist-test-slave-cq88 krb5kdc[13285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:55.674924 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:56.036326 13298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:56.036747 13298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:56.036926 13298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:56.044778 13298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:56.045003 13298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:56.045120 13298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:56.045234 13298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:56.045327 13298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:56.056102 13298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:56.057582 13298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:56.059031 13298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:56.072242 13304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:56.072289 13307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:56.072218 13305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:56.073289 13298 server_base.cc:1034] running on GCE node
I20260929 16:02:56.074231 13298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:56.075837 13298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:56.077100 13298 hybrid_clock.cc:648] HybridClock initialized: now 1790697776077036 us; error 61 us; skew 500 ppm
I20260929 16:02:56.077482 13298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:56 dist-test-slave-cq88 krb5kdc[13285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:56.084709 13298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:56.086483 13298 webserver.cc:460] Webserver started at http://127.0.0.1:32855/ using document root <none> and password file <none>
I20260929 16:02:56.087307 13298 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:56.087467 13298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:56.087797 13298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:56.091066 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "aad1dca55b064c16870740baf1f4abf7"
format_stamp: "Formatted at 2026-09-29 16:02:56 on dist-test-slave-cq88"
server_key: "7955fd9a9688f40be49d8bb114be781d"
server_key_iv: "5d7780b2630e33d78241ddad0bf4ae94"
server_key_version: "encryptionkey@0"
I20260929 16:02:56.091981 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "aad1dca55b064c16870740baf1f4abf7"
format_stamp: "Formatted at 2026-09-29 16:02:56 on dist-test-slave-cq88"
server_key: "7955fd9a9688f40be49d8bb114be781d"
server_key_iv: "5d7780b2630e33d78241ddad0bf4ae94"
server_key_version: "encryptionkey@0"
I20260929 16:02:56.097774 13298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:02:56.101696 13314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:56.103372 13298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:02:56.103603 13298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "aad1dca55b064c16870740baf1f4abf7"
format_stamp: "Formatted at 2026-09-29 16:02:56 on dist-test-slave-cq88"
server_key: "7955fd9a9688f40be49d8bb114be781d"
server_key_iv: "5d7780b2630e33d78241ddad0bf4ae94"
server_key_version: "encryptionkey@0"
I20260929 16:02:56.103847 13298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:56.122988 13298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:56.128311 13298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:56.128659 13298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:56.149430 13298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43121
I20260929 16:02:56.149533 13365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43121 every 8 connection(s)
I20260929 16:02:56.151015 13298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:56.152693 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 13298
I20260929 16:02:56.152902 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:56.153386 24441 external_mini_cluster.cc:1428] Setting key 537fd7b0bca2de21ceb7a19b3e945237
I20260929 16:02:56.155614 13366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:56.167959 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.155376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:43775 (local address 127.0.0.1:43121)
0929 16:02:56.156257 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:56.156268 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:56.156315 (+ 47us) server_negotiation.cc:407] Connection header received
0929 16:02:56.158044 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:56.158070 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:56.158522 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:56.158916 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:56.159622 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.161079 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.163056 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.163864 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.164011 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.165391 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:56.165401 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:56.165416 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:56.166023 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:56.166104 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:56.166130 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:56.166262 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:56.166631 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 16:02:56.166843 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":417,"threads_started":1}
I20260929 16:02:56.169112 13366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7: Bootstrap starting.
I20260929 16:02:56.172283 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:02:56.172664 13366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:56.173671 13366 log.cc:824] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:56.177099 13366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7: No bootstrap required, opened a new log
I20260929 16:02:56.189342 13366 raft_consensus.cc:348] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad1dca55b064c16870740baf1f4abf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43121 } }
I20260929 16:02:56.189899 13366 raft_consensus.cc:374] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:56.190023 13366 raft_consensus.cc:729] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aad1dca55b064c16870740baf1f4abf7, State: Initialized, Role: FOLLOWER
I20260929 16:02:56.190724 13366 consensus_queue.cc:260] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad1dca55b064c16870740baf1f4abf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43121 } }
I20260929 16:02:56.191056 13366 raft_consensus.cc:388] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:56.191161 13369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:56.177234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:41469 (local address 127.0.0.1:43121)
0929 16:02:56.177431 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:56.177437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:56.183413 (+ 5976us) server_negotiation.cc:407] Connection header received
0929 16:02:56.183840 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:56.183847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:56.183963 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:56.184083 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:56.184842 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.186212 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.188260 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:02:56.188861 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:02:56.188973 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:02:56.190282 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:56.190287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:56.190290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:56.190313 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:56.190352 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:56.190367 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:56.190506 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:56.190716 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 16:02:56.190774 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:02:56.191210 13366 raft_consensus.cc:482] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:56.191499 13366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:56.194047 13366 raft_consensus.cc:504] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad1dca55b064c16870740baf1f4abf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43121 } }
I20260929 16:02:56.194530 13366 leader_election.cc:302] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aad1dca55b064c16870740baf1f4abf7; no voters:
I20260929 16:02:56.195104 13366 leader_election.cc:288] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:56.195341 13376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:56.196902 13376 raft_consensus.cc:686] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [term 1 LEADER]: Becoming Leader. State: Replica: aad1dca55b064c16870740baf1f4abf7, State: Running, Role: LEADER
I20260929 16:02:56.197606 13366 sys_catalog.cc:564] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:56.197526 13376 consensus_queue.cc:237] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad1dca55b064c16870740baf1f4abf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43121 } }
I20260929 16:02:56.201236 13378 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aad1dca55b064c16870740baf1f4abf7. Latest consensus state: current_term: 1 leader_uuid: "aad1dca55b064c16870740baf1f4abf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad1dca55b064c16870740baf1f4abf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43121 } } }
I20260929 16:02:56.201347 13377 sys_catalog.cc:455] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aad1dca55b064c16870740baf1f4abf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aad1dca55b064c16870740baf1f4abf7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43121 } } }
I20260929 16:02:56.202031 13377 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:56.202028 13378 sys_catalog.cc:458] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:56.203068 13384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:56.211557 13384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:56.221280 13384 catalog_manager.cc:1347] Generated new cluster ID: ea1fd29708eb4562984e2c0d0e47028a
I20260929 16:02:56.221473 13384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:56.239569 13384 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:56.241544 13384 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:56.249831 13384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aad1dca55b064c16870740baf1f4abf7: Generated new TSK 0
I20260929 16:02:56.250792 13384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:56.265532 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13298
W20260929 16:02:56.267166 13361 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:43775
W20260929 16:02:56.268229 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13298: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:02:56.268392 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 13298
W20260929 16:02:56.287436 13283 connection.cc:535] client connection to 127.0.0.1:43121 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43121): Connection reset by peer (error 104)
2026-09-29T16:02:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2009 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:56 dist-test-slave-cq88 krb5kdc[13397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:56 dist-test-slave-cq88 krb5kdc[13397](info): set up 2 sockets
Sep 29 16:02:56 dist-test-slave-cq88 krb5kdc[13397](info): commencing operation
krb5kdc: starting...
W20260929 16:02:57.402832 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.008s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:57 dist-test-slave-cq88 krb5kdc[13397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:57.679550 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:58.019685 13410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:58.020089 13410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:58.020259 13410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:58.029484 13410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:58.029667 13410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:58.029755 13410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:58.029846 13410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:58.029940 13410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:58.040547 13410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:58.041980 13410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:58.043448 13410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:58.056309 13415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:58.056279 13416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:58.056633 13418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:58.057096 13410 server_base.cc:1034] running on GCE node
I20260929 16:02:58.057907 13410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:58.059440 13410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:58.060675 13410 hybrid_clock.cc:648] HybridClock initialized: now 1790697778060628 us; error 42 us; skew 500 ppm
I20260929 16:02:58.061055 13410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:58 dist-test-slave-cq88 krb5kdc[13397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:58.064376 13410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:58.065915 13410 webserver.cc:460] Webserver started at http://127.0.0.1:40189/ using document root <none> and password file <none>
I20260929 16:02:58.066630 13410 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:58.066783 13410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:58.067101 13410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:58.070207 13410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "f8bbc1ae355441038897a350776d9b2e"
format_stamp: "Formatted at 2026-09-29 16:02:58 on dist-test-slave-cq88"
server_key: "4f49c02bde57c787eb108a13e5939287"
server_key_iv: "6b9b44c0845da497b0baed8d783919f3"
server_key_version: "encryptionkey@0"
I20260929 16:02:58.071035 13410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "f8bbc1ae355441038897a350776d9b2e"
format_stamp: "Formatted at 2026-09-29 16:02:58 on dist-test-slave-cq88"
server_key: "4f49c02bde57c787eb108a13e5939287"
server_key_iv: "6b9b44c0845da497b0baed8d783919f3"
server_key_version: "encryptionkey@0"
I20260929 16:02:58.076527 13410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 16:02:58.080124 13425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:02:58.081532 13410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:02:58.081730 13410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "f8bbc1ae355441038897a350776d9b2e"
format_stamp: "Formatted at 2026-09-29 16:02:58 on dist-test-slave-cq88"
server_key: "4f49c02bde57c787eb108a13e5939287"
server_key_iv: "6b9b44c0845da497b0baed8d783919f3"
server_key_version: "encryptionkey@0"
I20260929 16:02:58.081960 13410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:02:58.088038 13410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:02:58.092162 13410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:02:58.092481 13410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:02:58.111483 13410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38635
I20260929 16:02:58.111572 13477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38635 every 8 connection(s)
I20260929 16:02:58.112896 13410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:02:58.116510 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 13410
I20260929 16:02:58.116696 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:02:58.116916 13478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:02:58.117259 24441 external_mini_cluster.cc:1428] Setting key 6563ea01f47dedadc13aa039cfb9b8ad
I20260929 16:02:58.125638 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:58.119125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:42879 (local address 127.0.0.1:38635)
0929 16:02:58.119950 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:58.119962 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:58.120006 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:02:58.121792 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:58.121816 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:58.122179 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:58.122623 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:58.123875 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:58.123882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:58.123896 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:58.123912 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:58.123977 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:58.124001 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:58.124265 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:58.124440 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 16:02:58.124666 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":346,"threads_started":1}
I20260929 16:02:58.129511 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:02:58.131286 13478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e: Bootstrap starting.
I20260929 16:02:58.135178 13478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e: Neither blocks nor log segments found. Creating new log.
I20260929 16:02:58.136842 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:02:58.133926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:54523 (local address 127.0.0.1:38635)
0929 16:02:58.134149 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 16:02:58.134158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:02:58.134674 (+ 516us) server_negotiation.cc:407] Connection header received
0929 16:02:58.134817 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:02:58.134824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:02:58.134918 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:02:58.135077 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:02:58.136106 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:02:58.136112 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:02:58.136117 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:02:58.136137 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:02:58.136190 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:02:58.136214 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:02:58.136372 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:02:58.136430 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 16:02:58.136501 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:02:58.136762 13478 log.cc:824] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:02:58.140033 13478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e: No bootstrap required, opened a new log
I20260929 16:02:58.147848 13478 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8bbc1ae355441038897a350776d9b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } }
I20260929 16:02:58.148249 13478 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:02:58.148339 13478 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8bbc1ae355441038897a350776d9b2e, State: Initialized, Role: FOLLOWER
I20260929 16:02:58.148880 13478 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8bbc1ae355441038897a350776d9b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } }
I20260929 16:02:58.149106 13478 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:02:58.149214 13478 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:02:58.149355 13478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:02:58.151814 13478 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8bbc1ae355441038897a350776d9b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } }
I20260929 16:02:58.152329 13478 leader_election.cc:302] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8bbc1ae355441038897a350776d9b2e; no voters:
I20260929 16:02:58.152896 13478 leader_election.cc:288] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:02:58.153059 13488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:02:58.153745 13488 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [term 1 LEADER]: Becoming Leader. State: Replica: f8bbc1ae355441038897a350776d9b2e, State: Running, Role: LEADER
I20260929 16:02:58.154291 13488 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8bbc1ae355441038897a350776d9b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } }
I20260929 16:02:58.155169 13478 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:02:58.157219 13489 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8bbc1ae355441038897a350776d9b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8bbc1ae355441038897a350776d9b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } } }
I20260929 16:02:58.157341 13490 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8bbc1ae355441038897a350776d9b2e. Latest consensus state: current_term: 1 leader_uuid: "f8bbc1ae355441038897a350776d9b2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8bbc1ae355441038897a350776d9b2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38635 } } }
I20260929 16:02:58.157785 13490 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:58.157653 13489 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e [sys.catalog]: This master's current role is: LEADER
I20260929 16:02:58.159785 13494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:02:58.165237 13494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:02:58.175392 13505 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:02:58.175704 13494 catalog_manager.cc:1347] Generated new cluster ID: 8948f0e414a84094b32420625ee7ba27
I20260929 16:02:58.176182 13494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:02:58.193252 13494 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:02:58.193773 13494 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:02:58.207211 13494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8bbc1ae355441038897a350776d9b2e: Generated new TSK 0
I20260929 16:02:58.208356 13494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:02:58.213275 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13410
W20260929 16:02:58.214534 13473 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:42879
W20260929 16:02:58.215395 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13410: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:02:58.215567 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 13410
2026-09-29T16:02:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1945 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:02:58 dist-test-slave-cq88 krb5kdc[13509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:02:58 dist-test-slave-cq88 krb5kdc[13509](info): set up 2 sockets
Sep 29 16:02:58 dist-test-slave-cq88 krb5kdc[13509](info): commencing operation
krb5kdc: starting...
W20260929 16:02:59.326699 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:02:59 dist-test-slave-cq88 krb5kdc[13509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:02:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:02:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:02:59.599833 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:02:59.948881 13522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:02:59.949219 13522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:02:59.949419 13522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:02:59.957221 13522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:02:59.957418 13522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:02:59.957541 13522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:02:59.957670 13522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:02:59.957742 13522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:02:59.968402 13522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:02:59.969836 13522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:02:59.971297 13522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:02:59.984254 13527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:59.984176 13528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:02:59.984202 13530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:02:59.985163 13522 server_base.cc:1034] running on GCE node
I20260929 16:02:59.986070 13522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:02:59.987778 13522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:02:59.989049 13522 hybrid_clock.cc:648] HybridClock initialized: now 1790697779989006 us; error 37 us; skew 500 ppm
I20260929 16:02:59.989414 13522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:02:59 dist-test-slave-cq88 krb5kdc[13509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:02:59.993027 13522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:02:59.994738 13522 webserver.cc:460] Webserver started at http://127.0.0.1:42605/ using document root <none> and password file <none>
I20260929 16:02:59.995576 13522 fs_manager.cc:359] Metadata directory not provided
I20260929 16:02:59.995729 13522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:02:59.996013 13522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:02:59.999068 13522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "531e4597a2424ceeb42d638f2abe8093"
format_stamp: "Formatted at 2026-09-29 16:02:59 on dist-test-slave-cq88"
server_key: "d2290814030a281052e7c1983196445c"
server_key_iv: "853123701e4ece817147fc21ffdbe1b8"
server_key_version: "encryptionkey@0"
I20260929 16:02:59.999950 13522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "531e4597a2424ceeb42d638f2abe8093"
format_stamp: "Formatted at 2026-09-29 16:02:59 on dist-test-slave-cq88"
server_key: "d2290814030a281052e7c1983196445c"
server_key_iv: "853123701e4ece817147fc21ffdbe1b8"
server_key_version: "encryptionkey@0"
I20260929 16:03:00.005414 13522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:03:00.009168 13537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:00.010993 13522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:03:00.011198 13522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "531e4597a2424ceeb42d638f2abe8093"
format_stamp: "Formatted at 2026-09-29 16:02:59 on dist-test-slave-cq88"
server_key: "d2290814030a281052e7c1983196445c"
server_key_iv: "853123701e4ece817147fc21ffdbe1b8"
server_key_version: "encryptionkey@0"
I20260929 16:03:00.011492 13522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:00.025557 13522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:00.031391 13522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:00.031863 13522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:00.050870 13522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44807
I20260929 16:03:00.050909 13589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44807 every 8 connection(s)
I20260929 16:03:00.052469 13522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:00.056772 13590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:00.057205 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 13522
I20260929 16:03:00.057338 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:00.057735 24441 external_mini_cluster.cc:1428] Setting key f803223e2920023a78cdebb21bbc6e76
I20260929 16:03:00.072016 13590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093: Bootstrap starting.
Sep 29 16:03:00 dist-test-slave-cq88 krb5kdc[13509](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 16:03:00.076283 13590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:00.077554 13590 log.cc:824] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:00.081298 13590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093: No bootstrap required, opened a new log
I20260929 16:03:00.085285 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.059790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:53625 (local address 127.0.0.1:44807)
0929 16:03:00.060642 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:00.060654 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:00.060696 (+ 42us) server_negotiation.cc:407] Connection header received
0929 16:03:00.063097 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:00.063124 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:00.063671 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:00.064029 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:00.066122 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.067576 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.069591 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.070341 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.070485 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.075822 (+ 5337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:00.075830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:00.075845 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:00.075890 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:00.079744 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.080733 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.080742 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.080751 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.081009 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.081542 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.081547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.081550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.082408 (+ 858us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:00.082747 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:00.083210 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 16:03:00.083640 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":412,"threads_started":1}
I20260929 16:03:00.090126 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:00.092145 13590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531e4597a2424ceeb42d638f2abe8093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260929 16:03:00.092602 13590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:00.092738 13590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531e4597a2424ceeb42d638f2abe8093, State: Initialized, Role: FOLLOWER
I20260929 16:03:00.093451 13590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531e4597a2424ceeb42d638f2abe8093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260929 16:03:00.093730 13590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:00.093909 13590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:00.094156 13590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:00.097239 13590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531e4597a2424ceeb42d638f2abe8093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260929 16:03:00.097973 13590 leader_election.cc:302] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531e4597a2424ceeb42d638f2abe8093; no voters:
I20260929 16:03:00.098722 13590 leader_election.cc:288] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:00.098814 13600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:00.101282 13590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:00.101943 13600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [term 1 LEADER]: Becoming Leader. State: Replica: 531e4597a2424ceeb42d638f2abe8093, State: Running, Role: LEADER
I20260929 16:03:00.103003 13600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531e4597a2424ceeb42d638f2abe8093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } }
I20260929 16:03:00.105646 13602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531e4597a2424ceeb42d638f2abe8093. Latest consensus state: current_term: 1 leader_uuid: "531e4597a2424ceeb42d638f2abe8093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531e4597a2424ceeb42d638f2abe8093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260929 16:03:00.106050 13602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:00.106635 13601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531e4597a2424ceeb42d638f2abe8093" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531e4597a2424ceeb42d638f2abe8093" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44807 } } }
I20260929 16:03:00.107008 13601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:00.124403 13605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:00.130326 13605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:00.130339 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:00.094779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:55443 (local address 127.0.0.1:44807)
0929 16:03:00.095008 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:00.095016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:00.095503 (+ 487us) server_negotiation.cc:407] Connection header received
0929 16:03:00.095659 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:00.095666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:00.095760 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:00.095899 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:00.097388 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.098621 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.100812 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:00.101585 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:00.102486 (+ 901us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:00.106424 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:00.106440 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:00.106449 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:00.106514 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:00.124364 (+ 17850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.125814 (+ 1450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.125820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.125824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.125990 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:00.126796 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:00.126805 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:00.126811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:00.127450 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:00.127826 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:00.129459 (+ 1633us) server_negotiation.cc:299] Negotiation successful
0929 16:03:00.129729 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:03:00.141556 13605 catalog_manager.cc:1347] Generated new cluster ID: fdf7104baba646a098ff4aa35ec3d084
I20260929 16:03:00.141691 13605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:00.158078 13605 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:00.159482 13605 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:00.168363 13605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 531e4597a2424ceeb42d638f2abe8093: Generated new TSK 0
I20260929 16:03:00.169236 13605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:00.184412 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13522
I20260929 16:03:00.186149 13585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:00.281400 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 13522
W20260929 16:03:00.299377 13507 connection.cc:535] client connection to 127.0.0.1:44807 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44807): Connection reset by peer (error 104)
2026-09-29T16:03:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2070 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:00 dist-test-slave-cq88 krb5kdc[13623](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:00 dist-test-slave-cq88 krb5kdc[13623](info): set up 2 sockets
Sep 29 16:03:00 dist-test-slave-cq88 krb5kdc[13623](info): commencing operation
krb5kdc: starting...
W20260929 16:03:01.405521 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:01 dist-test-slave-cq88 krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:01.700963 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:02.048036 13637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:02.048358 13637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:02.048520 13637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:02.056268 13637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:02.056393 13637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:02.056506 13637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:02.056602 13637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:02.056676 13637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:02.066995 13637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:02.068430 13637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:02.069840 13637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:02.082545 13642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:02.082496 13643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:02.082466 13645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:02.083655 13637 server_base.cc:1034] running on GCE node
I20260929 16:03:02.084444 13637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:02.085978 13637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:02.087190 13637 hybrid_clock.cc:648] HybridClock initialized: now 1790697782087127 us; error 55 us; skew 500 ppm
I20260929 16:03:02.087654 13637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:02 dist-test-slave-cq88 krb5kdc[13623](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:02.091001 13637 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:02.092693 13637 webserver.cc:460] Webserver started at http://127.0.0.1:44401/ using document root <none> and password file <none>
I20260929 16:03:02.093418 13637 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:02.093533 13637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:02.093801 13637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:02.097023 13637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "c77a06e102b841fbbb35778d907a3f32"
format_stamp: "Formatted at 2026-09-29 16:03:02 on dist-test-slave-cq88"
server_key: "bc24601c3713adeb546913654982fd91"
server_key_iv: "975e63cf1d5c918cb588667b4a0242e4"
server_key_version: "encryptionkey@0"
I20260929 16:03:02.097767 13637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "c77a06e102b841fbbb35778d907a3f32"
format_stamp: "Formatted at 2026-09-29 16:03:02 on dist-test-slave-cq88"
server_key: "bc24601c3713adeb546913654982fd91"
server_key_iv: "975e63cf1d5c918cb588667b4a0242e4"
server_key_version: "encryptionkey@0"
I20260929 16:03:02.103485 13637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:03:02.107359 13652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:02.108791 13637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:02.109001 13637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "c77a06e102b841fbbb35778d907a3f32"
format_stamp: "Formatted at 2026-09-29 16:03:02 on dist-test-slave-cq88"
server_key: "bc24601c3713adeb546913654982fd91"
server_key_iv: "975e63cf1d5c918cb588667b4a0242e4"
server_key_version: "encryptionkey@0"
I20260929 16:03:02.109226 13637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:02.121765 13637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:02.126740 13637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:02.127089 13637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:02.147276 13637 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44547
I20260929 16:03:02.147302 13703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44547 every 8 connection(s)
I20260929 16:03:02.148775 13637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:02.152684 13704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:02.158290 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 13637
I20260929 16:03:02.158499 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:02.158916 24441 external_mini_cluster.cc:1428] Setting key 960e4a361d3987c17e43394f63a8d7bb
I20260929 16:03:02.165572 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32: Bootstrap starting.
I20260929 16:03:02.169616 13704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:02.170804 13704 log.cc:824] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:02.174652 13704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32: No bootstrap required, opened a new log
I20260929 16:03:02.175302 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:02.160978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:32913 (local address 127.0.0.1:44547)
0929 16:03:02.161906 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:02.161924 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:02.161977 (+ 53us) server_negotiation.cc:407] Connection header received
0929 16:03:02.164518 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:02.164555 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:02.165167 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:02.165658 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:02.166252 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:02.168347 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:02.169856 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:02.170608 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:02.170838 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:02.172274 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:02.172282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:02.172297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:02.173087 (+ 790us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:02.173199 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:02.173234 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:02.173409 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:02.173699 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 16:03:02.173894 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":391,"threads_started":1}
I20260929 16:03:02.180258 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:02.188832 13704 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77a06e102b841fbbb35778d907a3f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } }
I20260929 16:03:02.189447 13704 raft_consensus.cc:374] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:02.189556 13704 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77a06e102b841fbbb35778d907a3f32, State: Initialized, Role: FOLLOWER
I20260929 16:03:02.190572 13704 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77a06e102b841fbbb35778d907a3f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } }
I20260929 16:03:02.190889 13704 raft_consensus.cc:388] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:02.191056 13704 raft_consensus.cc:482] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:02.191311 13704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:02.195487 13704 raft_consensus.cc:504] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77a06e102b841fbbb35778d907a3f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } }
I20260929 16:03:02.196640 13704 leader_election.cc:302] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c77a06e102b841fbbb35778d907a3f32; no voters:
I20260929 16:03:02.197453 13704 leader_election.cc:288] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:02.197587 13714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:02.197674 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:02.189217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:54513 (local address 127.0.0.1:44547)
0929 16:03:02.189887 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:02.189893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:02.189915 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:03:02.190168 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:02.190174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:02.190282 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:02.190397 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:02.191079 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:02.192074 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:02.194176 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:02.194930 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:02.195088 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:02.196522 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:02.196527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:02.196531 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:02.196550 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:02.196607 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:02.196626 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:02.196767 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:02.197149 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 16:03:02.197217 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542}
I20260929 16:03:02.198949 13714 raft_consensus.cc:686] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [term 1 LEADER]: Becoming Leader. State: Replica: c77a06e102b841fbbb35778d907a3f32, State: Running, Role: LEADER
I20260929 16:03:02.199568 13714 consensus_queue.cc:237] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77a06e102b841fbbb35778d907a3f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } }
I20260929 16:03:02.200169 13704 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:02.203271 13716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77a06e102b841fbbb35778d907a3f32. Latest consensus state: current_term: 1 leader_uuid: "c77a06e102b841fbbb35778d907a3f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77a06e102b841fbbb35778d907a3f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } } }
I20260929 16:03:02.203290 13715 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c77a06e102b841fbbb35778d907a3f32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77a06e102b841fbbb35778d907a3f32" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44547 } } }
I20260929 16:03:02.203773 13716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:02.203773 13715 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:02.204854 13721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:02.211539 13721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:02.222429 13721 catalog_manager.cc:1347] Generated new cluster ID: 602196675ab0416199897da9c2df12f9
I20260929 16:03:02.222553 13721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:02.232438 13721 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:02.233870 13721 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:02.243448 13721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c77a06e102b841fbbb35778d907a3f32: Generated new TSK 0
I20260929 16:03:02.244338 13721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:02.251703 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13637
W20260929 16:03:02.253592 13699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:32913
W20260929 16:03:02.254506 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13637: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:02.254719 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 13637
W20260929 16:03:02.269387 13621 connection.cc:535] client connection to 127.0.0.1:44547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44547): Connection reset by peer (error 104)
2026-09-29T16:03:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:02 dist-test-slave-cq88 krb5kdc[13735](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:02 dist-test-slave-cq88 krb5kdc[13735](info): set up 2 sockets
Sep 29 16:03:02 dist-test-slave-cq88 krb5kdc[13735](info): commencing operation
krb5kdc: starting...
W20260929 16:03:03.362632 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:03 dist-test-slave-cq88 krb5kdc[13735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 16:03:03.631255 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44659
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:03.965219 13748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:03.965523 13748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:03.965696 13748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:03.972929 13748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:03.973037 13748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:03.973084 13748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:03.973172 13748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:03.973263 13748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:03.982977 13748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:03.984287 13748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:03.985594 13748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:03.997905 13756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:03.997921 13754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:03.998379 13753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:03.998608 13748 server_base.cc:1034] running on GCE node
I20260929 16:03:03.999550 13748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:04.001034 13748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:04.002276 13748 hybrid_clock.cc:648] HybridClock initialized: now 1790697784002233 us; error 59 us; skew 500 ppm
I20260929 16:03:04.002642 13748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:03 dist-test-slave-cq88 krb5kdc[13735](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:04.005769 13748 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 16:03:04.007159 13748 webserver.cc:460] Webserver started at http://127.0.0.1:41857/ using document root <none> and password file <none>
I20260929 16:03:04.007861 13748 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:04.007975 13748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:04.008236 13748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:04.011230 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "8c1461a461c74fb8a751eb85641994c7"
format_stamp: "Formatted at 2026-09-29 16:03:04 on dist-test-slave-cq88"
server_key: "e3a65125f908c39bbfa2c0aae8f622f8"
server_key_iv: "18df56cb48f288dabb587248aa466068"
server_key_version: "encryptionkey@0"
I20260929 16:03:04.012059 13748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "8c1461a461c74fb8a751eb85641994c7"
format_stamp: "Formatted at 2026-09-29 16:03:04 on dist-test-slave-cq88"
server_key: "e3a65125f908c39bbfa2c0aae8f622f8"
server_key_iv: "18df56cb48f288dabb587248aa466068"
server_key_version: "encryptionkey@0"
I20260929 16:03:04.017448 13748 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:03:04.020886 13763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:04.022189 13748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 16:03:04.022327 13748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "8c1461a461c74fb8a751eb85641994c7"
format_stamp: "Formatted at 2026-09-29 16:03:04 on dist-test-slave-cq88"
server_key: "e3a65125f908c39bbfa2c0aae8f622f8"
server_key_iv: "18df56cb48f288dabb587248aa466068"
server_key_version: "encryptionkey@0"
I20260929 16:03:04.022500 13748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:04.028209 13748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:04.032187 13748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:04.032430 13748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:04.051895 13748 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44659
I20260929 16:03:04.051942 13815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44659 every 8 connection(s)
I20260929 16:03:04.053316 13748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:04.056350 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 13748
I20260929 16:03:04.056537 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:04.057085 24441 external_mini_cluster.cc:1428] Setting key c98c7b0fd322e9b19588ea80c2dc08d2
I20260929 16:03:04.057471 13816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:04.065399 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.058877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:39921 (local address 127.0.0.1:44659)
0929 16:03:04.059681 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.059693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.059733 (+ 40us) server_negotiation.cc:407] Connection header received
0929 16:03:04.062045 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.062068 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.062414 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.062741 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:04.063835 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:04.063844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:04.063857 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:04.063874 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:04.063935 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:04.063959 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:04.064150 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.064250 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.064464 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":383,"threads_started":1}
I20260929 16:03:04.069579 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:04.071702 13816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7: Bootstrap starting.
I20260929 16:03:04.075850 13816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:04.077065 13816 log.cc:824] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:04.077517 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:04.074871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.109:44261 (local address 127.0.0.1:44659)
0929 16:03:04.075086 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:04.075093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:04.075573 (+ 480us) server_negotiation.cc:407] Connection header received
0929 16:03:04.075725 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:04.075733 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:04.075802 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:04.075925 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:04.076885 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:04.076890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:04.076894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:04.076913 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:04.076959 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:04.076978 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:04.077119 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:04.077169 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 16:03:04.077230 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 16:03:04.080304 13816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7: No bootstrap required, opened a new log
I20260929 16:03:04.088018 13816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1461a461c74fb8a751eb85641994c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44659 } }
I20260929 16:03:04.088405 13816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:04.088529 13816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1461a461c74fb8a751eb85641994c7, State: Initialized, Role: FOLLOWER
I20260929 16:03:04.089115 13816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1461a461c74fb8a751eb85641994c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44659 } }
I20260929 16:03:04.089361 13816 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:04.089517 13816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:04.089687 13816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:04.092129 13816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1461a461c74fb8a751eb85641994c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44659 } }
I20260929 16:03:04.092635 13816 leader_election.cc:302] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c1461a461c74fb8a751eb85641994c7; no voters:
I20260929 16:03:04.093223 13816 leader_election.cc:288] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:04.093379 13826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:04.094064 13826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1461a461c74fb8a751eb85641994c7, State: Running, Role: LEADER
I20260929 16:03:04.094671 13826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1461a461c74fb8a751eb85641994c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44659 } }
I20260929 16:03:04.095435 13816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:04.097461 13828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c1461a461c74fb8a751eb85641994c7. Latest consensus state: current_term: 1 leader_uuid: "8c1461a461c74fb8a751eb85641994c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1461a461c74fb8a751eb85641994c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44659 } } }
I20260929 16:03:04.097929 13828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:04.097677 13827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c1461a461c74fb8a751eb85641994c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c1461a461c74fb8a751eb85641994c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44659 } } }
I20260929 16:03:04.098102 13827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:04.100966 13833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:04.106369 13833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:04.115731 13833 catalog_manager.cc:1347] Generated new cluster ID: 25a4d5b8bf8940cda3310e02ca1097b1
I20260929 16:03:04.115854 13833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:04.132683 13833 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:04.133141 13833 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:04.139758 13833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c1461a461c74fb8a751eb85641994c7: Generated new TSK 0
I20260929 16:03:04.140585 13833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:04.152957 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13748
W20260929 16:03:04.154109 13811 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.109:39921
W20260929 16:03:04.154829 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13748: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:04.154992 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 13748
2026-09-29T16:03:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:04 dist-test-slave-cq88 krb5kdc[13847](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:04 dist-test-slave-cq88 krb5kdc[13847](info): set up 2 sockets
Sep 29 16:03:04 dist-test-slave-cq88 krb5kdc[13847](info): commencing operation
krb5kdc: starting...
W20260929 16:03:05.262569 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:05 dist-test-slave-cq88 krb5kdc[13847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:03:05.530611 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44031
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:44031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:05.868492 13860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:05.868860 13860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:05.869040 13860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:05.876528 13860 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:05.876720 13860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:05.876840 13860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:05.876955 13860 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:05.877046 13860 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:05.887320 13860 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:44031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:05.888827 13860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:05.890228 13860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:05.902940 13865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:05.902973 13868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:05.902884 13866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:05.904273 13860 server_base.cc:1034] running on GCE node
I20260929 16:03:05.905126 13860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:05.906669 13860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:05.907874 13860 hybrid_clock.cc:648] HybridClock initialized: now 1790697785907831 us; error 36 us; skew 500 ppm
I20260929 16:03:05.908252 13860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:05 dist-test-slave-cq88 krb5kdc[13847](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697785, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:05.911598 13860 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:03:05.913341 13860 webserver.cc:460] Webserver started at http://127.23.222.126:35439/ using document root <none> and password file <none>
I20260929 16:03:05.914109 13860 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:05.914271 13860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:05.914609 13860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:05.917893 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "284165bc7114473fa5c479fa0d148f3e"
format_stamp: "Formatted at 2026-09-29 16:03:05 on dist-test-slave-cq88"
server_key: "fadea6b67e36ab69837b0f8792aa2639"
server_key_iv: "9090c2d4577868843952f0af308ff5bd"
server_key_version: "encryptionkey@0"
I20260929 16:03:05.918727 13860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "284165bc7114473fa5c479fa0d148f3e"
format_stamp: "Formatted at 2026-09-29 16:03:05 on dist-test-slave-cq88"
server_key: "fadea6b67e36ab69837b0f8792aa2639"
server_key_iv: "9090c2d4577868843952f0af308ff5bd"
server_key_version: "encryptionkey@0"
I20260929 16:03:05.924293 13860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:03:05.928068 13875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:05.929422 13860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:05.929626 13860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "284165bc7114473fa5c479fa0d148f3e"
format_stamp: "Formatted at 2026-09-29 16:03:05 on dist-test-slave-cq88"
server_key: "fadea6b67e36ab69837b0f8792aa2639"
server_key_iv: "9090c2d4577868843952f0af308ff5bd"
server_key_version: "encryptionkey@0"
I20260929 16:03:05.929852 13860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:05.943120 13860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:05.947589 13860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:05.947888 13860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:05.967489 13860 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:44031
I20260929 16:03:05.967519 13927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:44031 every 8 connection(s)
I20260929 16:03:05.968947 13860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:05.972860 13928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:05.977185 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 13860
I20260929 16:03:05.977386 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:05.977957 24441 external_mini_cluster.cc:1428] Setting key d0f48c9c541c8143a95125adb8800c13
I20260929 16:03:05.984995 13928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e: Bootstrap starting.
I20260929 16:03:05.988340 13928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:05.989439 13928 log.cc:824] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:05.992589 13928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e: No bootstrap required, opened a new log
Sep 29 16:03:05 dist-test-slave-cq88 krb5kdc[13847](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:03:06.001317 13928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284165bc7114473fa5c479fa0d148f3e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44031 } }
I20260929 16:03:06.001840 13928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:06.002010 13928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284165bc7114473fa5c479fa0d148f3e, State: Initialized, Role: FOLLOWER
I20260929 16:03:06.002770 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:05.980151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55719 (local address 127.23.222.126:44031)
0929 16:03:05.981055 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:05.981067 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:05.981116 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:03:05.983654 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:05.983681 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:05.984112 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:05.984536 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:05.986118 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:05.987568 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:05.988918 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:05.989677 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:05.989866 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:05.994229 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:05.994238 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:05.994252 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:05.994299 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:05.997859 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:05.998717 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:05.998725 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:05.998732 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:05.998932 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:05.999484 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:05.999489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:05.999491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:06.000106 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:06.000463 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:06.001020 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 16:03:06.001369 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":445,"threads_started":1}
I20260929 16:03:06.002691 13928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284165bc7114473fa5c479fa0d148f3e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44031 } }
I20260929 16:03:06.003026 13928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:06.003212 13928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:06.003463 13928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:06.006472 13928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284165bc7114473fa5c479fa0d148f3e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44031 } }
I20260929 16:03:06.007125 13928 leader_election.cc:302] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 284165bc7114473fa5c479fa0d148f3e; no voters:
I20260929 16:03:06.007409 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:06.007894 13928 leader_election.cc:288] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:06.008051 13933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:06.010004 13928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:06.018983 13933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [term 1 LEADER]: Becoming Leader. State: Replica: 284165bc7114473fa5c479fa0d148f3e, State: Running, Role: LEADER
I20260929 16:03:06.019884 13933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284165bc7114473fa5c479fa0d148f3e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44031 } }
I20260929 16:03:06.023439 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "284165bc7114473fa5c479fa0d148f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284165bc7114473fa5c479fa0d148f3e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44031 } } }
I20260929 16:03:06.023926 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:06.024477 13935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 284165bc7114473fa5c479fa0d148f3e. Latest consensus state: current_term: 1 leader_uuid: "284165bc7114473fa5c479fa0d148f3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284165bc7114473fa5c479fa0d148f3e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44031 } } }
I20260929 16:03:06.024806 13935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e [sys.catalog]: This master's current role is: LEADER
W20260929 16:03:06.039920 13952 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:03:06.040123 13952 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:03:06.040259 13953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:06.042634 13953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:06.043011 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:06.021045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46547 (local address 127.23.222.126:44031)
0929 16:03:06.021255 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:06.021261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:06.021850 (+ 589us) server_negotiation.cc:407] Connection header received
0929 16:03:06.022100 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:06.022106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:06.022193 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:06.022329 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:06.024655 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:06.026133 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:06.028838 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:06.029805 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:06.029936 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:06.034556 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:06.034564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:06.034570 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:06.034626 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:06.038890 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:06.040110 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:06.040119 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:06.040123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:06.040302 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:06.041019 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:06.041025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:06.041028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:06.041589 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:06.041905 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:06.042365 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 16:03:06.042599 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:03:06.053005 13953 catalog_manager.cc:1347] Generated new cluster ID: 9ac727eb0fd9479780e70b2336440cee
I20260929 16:03:06.053138 13953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:06.078076 13953 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:06.080004 13953 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:06.087730 13953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 284165bc7114473fa5c479fa0d148f3e: Generated new TSK 0
I20260929 16:03:06.088871 13953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:06.100134 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13860
I20260929 16:03:06.101663 13923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:06.198486 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 13860
W20260929 16:03:06.216209 13845 connection.cc:535] client connection to 127.23.222.126:44031 recv error: Network error: failed to read from TLS socket (remote: 127.23.222.126:44031): Connection reset by peer (error 104)
2026-09-29T16:03:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:06 dist-test-slave-cq88 krb5kdc[13960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:06 dist-test-slave-cq88 krb5kdc[13960](info): set up 2 sockets
Sep 29 16:03:06 dist-test-slave-cq88 krb5kdc[13960](info): commencing operation
krb5kdc: starting...
W20260929 16:03:07.318790 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:07 dist-test-slave-cq88 krb5kdc[13960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:03:07.591457 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:40367
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:40367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:07.935002 13973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:07.935442 13973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:07.935693 13973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:07.943809 13973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:07.943933 13973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:07.943979 13973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:07.944033 13973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:07.944077 13973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:07.954301 13973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:40367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:40367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:07.955734 13973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:07.957119 13973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:07.970124 13978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:07.970136 13979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:07.970420 13981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:07.970976 13973 server_base.cc:1034] running on GCE node
I20260929 16:03:07.971889 13973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:07.973433 13973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:07.974646 13973 hybrid_clock.cc:648] HybridClock initialized: now 1790697787974604 us; error 61 us; skew 500 ppm
I20260929 16:03:07.975033 13973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:07 dist-test-slave-cq88 krb5kdc[13960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697787, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:07.987772 13973 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:03:07.990135 13973 webserver.cc:460] Webserver started at http://127.23.222.126:33447/ using document root <none> and password file <none>
I20260929 16:03:07.991204 13973 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:07.991423 13973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:07.991874 13973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:07.996752 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "24ef97f41c894f7782e4bc1fde79140d"
format_stamp: "Formatted at 2026-09-29 16:03:07 on dist-test-slave-cq88"
server_key: "8a64e7bb7eb2dee7e6342e7dfec972e4"
server_key_iv: "deff974e8dcb853104a37182995baf2e"
server_key_version: "encryptionkey@0"
I20260929 16:03:07.997915 13973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "24ef97f41c894f7782e4bc1fde79140d"
format_stamp: "Formatted at 2026-09-29 16:03:07 on dist-test-slave-cq88"
server_key: "8a64e7bb7eb2dee7e6342e7dfec972e4"
server_key_iv: "deff974e8dcb853104a37182995baf2e"
server_key_version: "encryptionkey@0"
I20260929 16:03:08.005920 13973 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 16:03:08.011864 13988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:08.014040 13973 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:03:08.014322 13973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "24ef97f41c894f7782e4bc1fde79140d"
format_stamp: "Formatted at 2026-09-29 16:03:07 on dist-test-slave-cq88"
server_key: "8a64e7bb7eb2dee7e6342e7dfec972e4"
server_key_iv: "deff974e8dcb853104a37182995baf2e"
server_key_version: "encryptionkey@0"
I20260929 16:03:08.014820 13973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:08.035497 13973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:08.040076 13973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:08.040382 13973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:08.060491 13973 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:40367
I20260929 16:03:08.060500 14039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:40367 every 8 connection(s)
I20260929 16:03:08.062703 13973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:08.068606 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 13973
I20260929 16:03:08.068830 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:08.069451 24441 external_mini_cluster.cc:1428] Setting key a04ecd915498f4cdcc1e0457d4e358ce
I20260929 16:03:08.070346 14040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:08.083384 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:08.071682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35747 (local address 127.23.222.126:40367)
0929 16:03:08.072353 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:08.072365 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:08.072409 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:03:08.074047 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:08.074069 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:08.074465 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:08.074798 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:08.075455 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:08.076783 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:08.078642 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:08.079401 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:08.079532 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:08.080927 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:08.080934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:08.080948 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:08.081508 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:08.081587 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:08.081614 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:08.081801 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:08.082085 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 16:03:08.082250 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":266,"threads_started":1}
I20260929 16:03:08.085042 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d: Bootstrap starting.
I20260929 16:03:08.087783 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:08.088713 14040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:08.089722 14040 log.cc:824] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:08.092712 14040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d: No bootstrap required, opened a new log
I20260929 16:03:08.099658 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:08.092531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55525 (local address 127.23.222.126:40367)
0929 16:03:08.092723 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:08.092732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:08.093108 (+ 376us) server_negotiation.cc:407] Connection header received
0929 16:03:08.093290 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:08.093296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:08.093384 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:08.093514 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:08.094005 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:08.095215 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:08.096686 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:08.097373 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:08.097510 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:08.098746 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:08.098753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:08.098758 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:08.098780 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:08.098833 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:08.098855 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:08.098982 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:08.099186 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 16:03:08.099290 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:03:08.102674 14040 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef97f41c894f7782e4bc1fde79140d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40367 } }
I20260929 16:03:08.103091 14040 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:08.103210 14040 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ef97f41c894f7782e4bc1fde79140d, State: Initialized, Role: FOLLOWER
I20260929 16:03:08.103950 14040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef97f41c894f7782e4bc1fde79140d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40367 } }
I20260929 16:03:08.104205 14040 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:08.104342 14040 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:08.104557 14040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:08.106799 14040 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef97f41c894f7782e4bc1fde79140d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40367 } }
I20260929 16:03:08.107402 14040 leader_election.cc:302] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24ef97f41c894f7782e4bc1fde79140d; no voters:
I20260929 16:03:08.108105 14040 leader_election.cc:288] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:08.108263 14050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:08.109056 14050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [term 1 LEADER]: Becoming Leader. State: Replica: 24ef97f41c894f7782e4bc1fde79140d, State: Running, Role: LEADER
I20260929 16:03:08.109660 14050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef97f41c894f7782e4bc1fde79140d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40367 } }
I20260929 16:03:08.110313 14040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:08.112854 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24ef97f41c894f7782e4bc1fde79140d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef97f41c894f7782e4bc1fde79140d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40367 } } }
I20260929 16:03:08.113224 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:08.113679 14052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24ef97f41c894f7782e4bc1fde79140d. Latest consensus state: current_term: 1 leader_uuid: "24ef97f41c894f7782e4bc1fde79140d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24ef97f41c894f7782e4bc1fde79140d" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 40367 } } }
I20260929 16:03:08.113996 14052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:08.115648 14058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:08.121269 14058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:08.131402 14058 catalog_manager.cc:1347] Generated new cluster ID: 01cd3b70dd674418891ccc867ff53fa9
I20260929 16:03:08.131543 14058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:08.161895 14058 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:08.163995 14058 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:08.173789 14058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24ef97f41c894f7782e4bc1fde79140d: Generated new TSK 0
I20260929 16:03:08.174723 14058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:08.182757 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13973
W20260929 16:03:08.184808 14035 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35747
W20260929 16:03:08.185778 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 13973: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:08.185941 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 13973
W20260929 16:03:08.204353 13958 connection.cc:535] client connection to 127.23.222.126:40367 recv error: Network error: failed to read from TLS socket (remote: 127.23.222.126:40367): Connection reset by peer (error 104)
2026-09-29T16:03:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:08 dist-test-slave-cq88 krb5kdc[14072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:08 dist-test-slave-cq88 krb5kdc[14072](info): set up 2 sockets
Sep 29 16:03:08 dist-test-slave-cq88 krb5kdc[14072](info): commencing operation
krb5kdc: starting...
W20260929 16:03:09.298863 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:09 dist-test-slave-cq88 krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:03:09.575013 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:41693
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:44739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:41693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:09.923822 14085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:09.924199 14085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:09.924377 14085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:09.932126 14085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:09.932300 14085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:09.932408 14085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:09.932528 14085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:09.932616 14085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:09.943516 14085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:44739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:41693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:41693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:09.945006 14085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:09.946417 14085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:09.965631 14093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:09.965880 14090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:09.965692 14091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:09.966984 14085 server_base.cc:1034] running on GCE node
I20260929 16:03:09.967869 14085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:09.969384 14085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:09.970682 14085 hybrid_clock.cc:648] HybridClock initialized: now 1790697789970599 us; error 62 us; skew 500 ppm
I20260929 16:03:09.971194 14085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:09 dist-test-slave-cq88 krb5kdc[14072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697789, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:09.974802 14085 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:03:09.977241 14085 webserver.cc:460] Webserver started at http://127.23.222.126:35783/ using document root <none> and password file <none>
I20260929 16:03:09.977954 14085 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:09.978067 14085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:09.978322 14085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:09.981552 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "e3c99a0cd9a74ee7a83cd64db775be99"
format_stamp: "Formatted at 2026-09-29 16:03:09 on dist-test-slave-cq88"
server_key: "67c8a6528bc229213bcf71eaaeeef9ea"
server_key_iv: "80aed9bcb3fcea7eebc0bbf903cdea96"
server_key_version: "encryptionkey@0"
I20260929 16:03:09.982311 14085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "e3c99a0cd9a74ee7a83cd64db775be99"
format_stamp: "Formatted at 2026-09-29 16:03:09 on dist-test-slave-cq88"
server_key: "67c8a6528bc229213bcf71eaaeeef9ea"
server_key_iv: "80aed9bcb3fcea7eebc0bbf903cdea96"
server_key_version: "encryptionkey@0"
I20260929 16:03:09.987893 14085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:03:09.991847 14100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:09.993510 14085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 16:03:09.993769 14085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "e3c99a0cd9a74ee7a83cd64db775be99"
format_stamp: "Formatted at 2026-09-29 16:03:09 on dist-test-slave-cq88"
server_key: "67c8a6528bc229213bcf71eaaeeef9ea"
server_key_iv: "80aed9bcb3fcea7eebc0bbf903cdea96"
server_key_version: "encryptionkey@0"
I20260929 16:03:09.994033 14085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:10.000720 14085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:10.005549 14085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:10.005880 14085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:10.025940 14085 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:41693
I20260929 16:03:10.025977 14151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:41693 every 8 connection(s)
I20260929 16:03:10.027578 14085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:10.031581 14152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:10.032471 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 14085
I20260929 16:03:10.032658 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:10.033141 24441 external_mini_cluster.cc:1428] Setting key 4de28c78a1e8030b11e55bc084c4d3c0
I20260929 16:03:10.042985 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:10.035178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39887 (local address 127.23.222.126:41693)
0929 16:03:10.036172 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:10.036188 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:10.036242 (+ 54us) server_negotiation.cc:407] Connection header received
0929 16:03:10.038774 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:10.038807 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:10.039353 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:10.039899 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:10.040944 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:10.040953 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:10.040970 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:10.040987 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:10.041063 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:10.041095 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:10.041321 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:10.041448 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 16:03:10.041658 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":393,"threads_started":1}
I20260929 16:03:10.046947 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99: Bootstrap starting.
I20260929 16:03:10.047112 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:10.050341 14152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:10.051685 14152 log.cc:824] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:10.054441 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:10.051756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34831 (local address 127.23.222.126:41693)
0929 16:03:10.051968 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:10.051978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:10.052398 (+ 420us) server_negotiation.cc:407] Connection header received
0929 16:03:10.052630 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:10.052635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:10.052688 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:10.052806 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:10.053772 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:10.053778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:10.053782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:10.053801 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:10.053849 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:10.053865 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:10.054000 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:10.054105 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 16:03:10.054168 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:03:10.055171 14152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99: No bootstrap required, opened a new log
I20260929 16:03:10.063014 14152 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c99a0cd9a74ee7a83cd64db775be99" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41693 } }
I20260929 16:03:10.063524 14152 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:10.063649 14152 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c99a0cd9a74ee7a83cd64db775be99, State: Initialized, Role: FOLLOWER
I20260929 16:03:10.064325 14152 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c99a0cd9a74ee7a83cd64db775be99" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41693 } }
I20260929 16:03:10.064554 14152 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:10.064719 14152 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:10.064904 14152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:10.067150 14152 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c99a0cd9a74ee7a83cd64db775be99" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41693 } }
I20260929 16:03:10.067731 14152 leader_election.cc:302] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c99a0cd9a74ee7a83cd64db775be99; no voters:
I20260929 16:03:10.068455 14152 leader_election.cc:288] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:10.068567 14162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:10.069818 14162 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [term 1 LEADER]: Becoming Leader. State: Replica: e3c99a0cd9a74ee7a83cd64db775be99, State: Running, Role: LEADER
I20260929 16:03:10.070433 14162 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c99a0cd9a74ee7a83cd64db775be99" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41693 } }
I20260929 16:03:10.071094 14152 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:10.074965 14163 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c99a0cd9a74ee7a83cd64db775be99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c99a0cd9a74ee7a83cd64db775be99" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41693 } } }
I20260929 16:03:10.075611 14163 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:10.076269 14164 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c99a0cd9a74ee7a83cd64db775be99. Latest consensus state: current_term: 1 leader_uuid: "e3c99a0cd9a74ee7a83cd64db775be99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c99a0cd9a74ee7a83cd64db775be99" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 41693 } } }
I20260929 16:03:10.076642 14164 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:10.078137 14169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:10.083590 14169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:10.092908 14169 catalog_manager.cc:1347] Generated new cluster ID: fc697c3f8e0f4780ae3b8755dce60b0a
I20260929 16:03:10.093053 14169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:10.110997 14169 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:10.111552 14169 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:10.121656 14169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c99a0cd9a74ee7a83cd64db775be99: Generated new TSK 0
I20260929 16:03:10.122828 14169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:10.135386 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14085
W20260929 16:03:10.136723 14147 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39887
W20260929 16:03:10.137490 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14085: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:10.137655 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 14085
2026-09-29T16:03:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:10 dist-test-slave-cq88 krb5kdc[14183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:10 dist-test-slave-cq88 krb5kdc[14183](info): set up 2 sockets
Sep 29 16:03:10 dist-test-slave-cq88 krb5kdc[14183](info): commencing operation
krb5kdc: starting...
W20260929 16:03:11.258737 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:11 dist-test-slave-cq88 krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:03:11.550383 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44827
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:44827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:11.896955 14196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:11.897330 14196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:11.897512 14196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:11.904947 14196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:11.905114 14196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:11.905224 14196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:11.905334 14196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:11.905431 14196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:11.915714 14196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:44827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:11.917163 14196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:11.918557 14196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:11.931025 14201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:11.931116 14202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:11.931265 14204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:11.932020 14196 server_base.cc:1034] running on GCE node
I20260929 16:03:11.932862 14196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:11.934439 14196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:11.935664 14196 hybrid_clock.cc:648] HybridClock initialized: now 1790697791935625 us; error 62 us; skew 500 ppm
I20260929 16:03:11.936044 14196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:11 dist-test-slave-cq88 krb5kdc[14183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697791, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:11.939342 14196 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:03:11.941021 14196 webserver.cc:460] Webserver started at http://127.23.222.126:33779/ using document root <none> and password file <none>
I20260929 16:03:11.941752 14196 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:11.941903 14196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:11.942226 14196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:11.945544 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "448304de16b446c1803936cc088f0686"
format_stamp: "Formatted at 2026-09-29 16:03:11 on dist-test-slave-cq88"
server_key: "ace0df64fcbb4cf93adeb1c0673b9f58"
server_key_iv: "fbd954846bf81377d06a8a463b8a198d"
server_key_version: "encryptionkey@0"
I20260929 16:03:11.946389 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "448304de16b446c1803936cc088f0686"
format_stamp: "Formatted at 2026-09-29 16:03:11 on dist-test-slave-cq88"
server_key: "ace0df64fcbb4cf93adeb1c0673b9f58"
server_key_iv: "fbd954846bf81377d06a8a463b8a198d"
server_key_version: "encryptionkey@0"
I20260929 16:03:11.951929 14196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 16:03:11.955551 14211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:11.956892 14196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:11.957093 14196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "448304de16b446c1803936cc088f0686"
format_stamp: "Formatted at 2026-09-29 16:03:11 on dist-test-slave-cq88"
server_key: "ace0df64fcbb4cf93adeb1c0673b9f58"
server_key_iv: "fbd954846bf81377d06a8a463b8a198d"
server_key_version: "encryptionkey@0"
I20260929 16:03:11.957311 14196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:11.981052 14196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:11.985826 14196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:11.986150 14196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:12.005932 14196 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:44827
I20260929 16:03:12.005954 14264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:44827 every 8 connection(s)
I20260929 16:03:12.007560 14196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:12.011719 14265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:12.017817 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 14196
I20260929 16:03:12.018023 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:12.018544 24441 external_mini_cluster.cc:1428] Setting key 86caf54ed69166d310f49bea4d11b572
I20260929 16:03:12.024746 14265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686: Bootstrap starting.
I20260929 16:03:12.028973 14265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:12.030408 14265 log.cc:824] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:12.034643 14265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686: No bootstrap required, opened a new log
Sep 29 16:03:12 dist-test-slave-cq88 krb5kdc[14183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:03:12.046047 14265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448304de16b446c1803936cc088f0686" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44827 } }
I20260929 16:03:12.046569 14265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:12.046692 14265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448304de16b446c1803936cc088f0686, State: Initialized, Role: FOLLOWER
I20260929 16:03:12.047494 14265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448304de16b446c1803936cc088f0686" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44827 } }
I20260929 16:03:12.047780 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:12.020873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48967 (local address 127.23.222.126:44827)
0929 16:03:12.021856 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:12.021870 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:12.021921 (+ 51us) server_negotiation.cc:407] Connection header received
0929 16:03:12.023979 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:12.024011 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:12.024589 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:12.025082 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:12.027034 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:12.028531 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:12.030577 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:12.031810 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:12.032076 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:12.037482 (+ 5406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:12.037496 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:12.037523 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:12.037604 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:12.042244 (+ 4640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:12.043159 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:12.043170 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:12.043180 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:12.043485 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:12.043904 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:12.043909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:12.043913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:12.044772 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:12.045092 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:12.045807 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 16:03:12.046336 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":447,"threads_started":1}
I20260929 16:03:12.047801 14265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:12.048013 14265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:12.048233 14265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:12.051564 14265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448304de16b446c1803936cc088f0686" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44827 } }
I20260929 16:03:12.052222 14265 leader_election.cc:302] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 448304de16b446c1803936cc088f0686; no voters:
I20260929 16:03:12.052317 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:12.052865 14265 leader_election.cc:288] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:12.053022 14270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:12.053653 14270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [term 1 LEADER]: Becoming Leader. State: Replica: 448304de16b446c1803936cc088f0686, State: Running, Role: LEADER
I20260929 16:03:12.054330 14270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448304de16b446c1803936cc088f0686" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44827 } }
I20260929 16:03:12.057364 14265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:12.058059 14272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "448304de16b446c1803936cc088f0686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448304de16b446c1803936cc088f0686" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44827 } } }
I20260929 16:03:12.058614 14272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:12.058085 14271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 448304de16b446c1803936cc088f0686. Latest consensus state: current_term: 1 leader_uuid: "448304de16b446c1803936cc088f0686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "448304de16b446c1803936cc088f0686" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44827 } } }
I20260929 16:03:12.059561 14271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:12.068081 14276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:12.076576 14276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:12.090746 14276 catalog_manager.cc:1347] Generated new cluster ID: 0668b9b7436f4ba2b1459cc2a3c87653
I20260929 16:03:12.090891 14276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:12.093720 14268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:12.070697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38983 (local address 127.23.222.126:44827)
0929 16:03:12.070909 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:12.070919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:12.071589 (+ 670us) server_negotiation.cc:407] Connection header received
0929 16:03:12.071812 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:12.071819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:12.071940 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:12.072072 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:12.074278 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:12.075542 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:12.077520 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:12.078264 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:12.078387 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:12.085901 (+ 7514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:12.085911 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:12.085916 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:12.085964 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:12.090154 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:12.091158 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:12.091164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:12.091167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:12.091355 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:12.091839 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:12.091844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:12.091848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:12.092456 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:12.092715 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:12.093125 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 16:03:12.093356 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 16:03:12.137655 14276 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:12.139163 14276 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:12.149849 14276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 448304de16b446c1803936cc088f0686: Generated new TSK 0
I20260929 16:03:12.150756 14276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:12.165366 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14196
I20260929 16:03:12.166757 14260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:12.266225 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 14196
2026-09-29T16:03:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2132 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:12 dist-test-slave-cq88 krb5kdc[14298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:12 dist-test-slave-cq88 krb5kdc[14298](info): set up 2 sockets
Sep 29 16:03:12 dist-test-slave-cq88 krb5kdc[14298](info): commencing operation
krb5kdc: starting...
W20260929 16:03:13.382704 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:13 dist-test-slave-cq88 krb5kdc[14298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:03:13.651104 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:39509
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:39509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:13.983219 14311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:13.983539 14311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:13.983734 14311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:13.991110 14311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:13.991261 14311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:13.991381 14311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:13.991449 14311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:13.991496 14311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:14.001654 14311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:39509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:39509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:14.002969 14311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:14.004282 14311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:14.016988 14316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:14.016914 14319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:14.016964 14317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:14.017836 14311 server_base.cc:1034] running on GCE node
I20260929 16:03:14.018682 14311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:14.020438 14311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:14.021709 14311 hybrid_clock.cc:648] HybridClock initialized: now 1790697794021659 us; error 41 us; skew 500 ppm
I20260929 16:03:14.022150 14311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:14 dist-test-slave-cq88 krb5kdc[14298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697794, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:14.032545 14311 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:03:14.034385 14311 webserver.cc:460] Webserver started at http://127.23.222.126:44255/ using document root <none> and password file <none>
I20260929 16:03:14.035197 14311 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:14.035418 14311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:14.035763 14311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:14.039008 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "9de861f0ecd04f649bc8cf503327c7b8"
format_stamp: "Formatted at 2026-09-29 16:03:14 on dist-test-slave-cq88"
server_key: "9a3da136303e8d2fd626cf0bed97996c"
server_key_iv: "7b7a068208726e462d8cfc7e00396017"
server_key_version: "encryptionkey@0"
I20260929 16:03:14.039917 14311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "9de861f0ecd04f649bc8cf503327c7b8"
format_stamp: "Formatted at 2026-09-29 16:03:14 on dist-test-slave-cq88"
server_key: "9a3da136303e8d2fd626cf0bed97996c"
server_key_iv: "7b7a068208726e462d8cfc7e00396017"
server_key_version: "encryptionkey@0"
I20260929 16:03:14.045398 14311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:03:14.049319 14326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:14.050884 14311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:03:14.051100 14311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "9de861f0ecd04f649bc8cf503327c7b8"
format_stamp: "Formatted at 2026-09-29 16:03:14 on dist-test-slave-cq88"
server_key: "9a3da136303e8d2fd626cf0bed97996c"
server_key_iv: "7b7a068208726e462d8cfc7e00396017"
server_key_version: "encryptionkey@0"
I20260929 16:03:14.051389 14311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:14.070986 14311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:14.075318 14311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:14.075652 14311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:14.095916 14311 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:39509
I20260929 16:03:14.095935 14378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:39509 every 8 connection(s)
I20260929 16:03:14.097359 14311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:14.101110 14379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:14.107755 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 14311
I20260929 16:03:14.107937 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:14.108491 24441 external_mini_cluster.cc:1428] Setting key b0178b1c1a14a705fc0ce521c7bdb346
I20260929 16:03:14.112273 14379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8: Bootstrap starting.
I20260929 16:03:14.115752 14379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:14.116814 14379 log.cc:824] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:14.120069 14379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8: No bootstrap required, opened a new log
I20260929 16:03:14.122409 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:14.110339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56803 (local address 127.23.222.126:39509)
0929 16:03:14.111142 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:14.111155 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:14.111194 (+ 39us) server_negotiation.cc:407] Connection header received
0929 16:03:14.113012 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:14.113035 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:14.113406 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:14.113739 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:14.114378 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:14.115792 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:14.117329 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:14.118169 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:14.118361 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:14.119525 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:14.119538 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:14.119557 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:14.120262 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:14.120360 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:14.120390 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:14.120529 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:14.120675 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 16:03:14.120951 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":379,"threads_started":1}
I20260929 16:03:14.126302 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:14.129930 14379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de861f0ecd04f649bc8cf503327c7b8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39509 } }
I20260929 16:03:14.130479 14379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:14.130614 14379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9de861f0ecd04f649bc8cf503327c7b8, State: Initialized, Role: FOLLOWER
I20260929 16:03:14.131311 14379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de861f0ecd04f649bc8cf503327c7b8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39509 } }
I20260929 16:03:14.131661 14379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:14.131837 14379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:14.132102 14379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:14.135432 14379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de861f0ecd04f649bc8cf503327c7b8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39509 } }
I20260929 16:03:14.136132 14379 leader_election.cc:302] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9de861f0ecd04f649bc8cf503327c7b8; no voters:
I20260929 16:03:14.136859 14379 leader_election.cc:288] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:14.136972 14389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:14.138927 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:14.131712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57757 (local address 127.23.222.126:39509)
0929 16:03:14.131938 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:14.131947 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:14.132417 (+ 470us) server_negotiation.cc:407] Connection header received
0929 16:03:14.132623 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:14.132627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:14.132688 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:14.132774 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:14.133390 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:14.134132 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:14.136001 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:14.136579 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:14.136756 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:14.138009 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:14.138014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:14.138017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:14.138035 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:14.138074 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:14.138090 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:14.138191 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:14.138611 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 16:03:14.138684 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:03:14.139670 14379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:14.143529 14389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [term 1 LEADER]: Becoming Leader. State: Replica: 9de861f0ecd04f649bc8cf503327c7b8, State: Running, Role: LEADER
I20260929 16:03:14.144346 14389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de861f0ecd04f649bc8cf503327c7b8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39509 } }
I20260929 16:03:14.147871 14390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9de861f0ecd04f649bc8cf503327c7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de861f0ecd04f649bc8cf503327c7b8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39509 } } }
I20260929 16:03:14.148361 14390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:14.149161 14391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9de861f0ecd04f649bc8cf503327c7b8. Latest consensus state: current_term: 1 leader_uuid: "9de861f0ecd04f649bc8cf503327c7b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9de861f0ecd04f649bc8cf503327c7b8" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 39509 } } }
I20260929 16:03:14.149549 14391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8 [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:14.150794 14398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:14.156680 14398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:14.166345 14398 catalog_manager.cc:1347] Generated new cluster ID: 65109b20ff944586951361ebcd27459b
I20260929 16:03:14.166481 14398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:14.196877 14398 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:14.198946 14398 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:14.207542 14398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9de861f0ecd04f649bc8cf503327c7b8: Generated new TSK 0
I20260929 16:03:14.208642 14398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:14.214902 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14311
W20260929 16:03:14.216337 14374 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56803
W20260929 16:03:14.217141 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14311: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:14.217308 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 14311
2026-09-29T16:03:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:14 dist-test-slave-cq88 krb5kdc[14410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:14 dist-test-slave-cq88 krb5kdc[14410](info): set up 2 sockets
Sep 29 16:03:14 dist-test-slave-cq88 krb5kdc[14410](info): commencing operation
krb5kdc: starting...
W20260929 16:03:15.335148 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:15 dist-test-slave-cq88 krb5kdc[14410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:03:15.611680 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44331
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:44331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:15.952523 14423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:15.952929 14423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:15.953104 14423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:15.960840 14423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:15.961033 14423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:15.961148 14423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:15.961262 14423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:15.961347 14423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:15.971621 14423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:44331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:15.973038 14423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:15.974406 14423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:15.986745 14428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:15.986770 14429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:15.986837 14432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:15.987615 14423 server_base.cc:1034] running on GCE node
I20260929 16:03:15.988412 14423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:15.989935 14423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:15.991281 14423 hybrid_clock.cc:648] HybridClock initialized: now 1790697795991179 us; error 42 us; skew 500 ppm
I20260929 16:03:15.991710 14423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:15 dist-test-slave-cq88 krb5kdc[14410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697795, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:15.995380 14423 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:03:15.997156 14423 webserver.cc:460] Webserver started at http://127.23.222.126:41133/ using document root <none> and password file <none>
I20260929 16:03:15.997876 14423 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:15.997987 14423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:15.998253 14423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:16.001508 14423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "205d935f053142b5b0b3dde49e74960b"
format_stamp: "Formatted at 2026-09-29 16:03:15 on dist-test-slave-cq88"
server_key: "3454e56e79e657ca6d58c0e67798d6d2"
server_key_iv: "793d3f8866d8f2228bddfbe81b5f3c9b"
server_key_version: "encryptionkey@0"
I20260929 16:03:16.002327 14423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "205d935f053142b5b0b3dde49e74960b"
format_stamp: "Formatted at 2026-09-29 16:03:15 on dist-test-slave-cq88"
server_key: "3454e56e79e657ca6d58c0e67798d6d2"
server_key_iv: "793d3f8866d8f2228bddfbe81b5f3c9b"
server_key_version: "encryptionkey@0"
I20260929 16:03:16.008154 14423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 16:03:16.012036 14439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:16.013562 14423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 16:03:16.013769 14423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "205d935f053142b5b0b3dde49e74960b"
format_stamp: "Formatted at 2026-09-29 16:03:15 on dist-test-slave-cq88"
server_key: "3454e56e79e657ca6d58c0e67798d6d2"
server_key_iv: "793d3f8866d8f2228bddfbe81b5f3c9b"
server_key_version: "encryptionkey@0"
I20260929 16:03:16.013998 14423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:16.020570 14423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:16.025331 14423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:16.025678 14423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:16.045022 14423 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:44331
I20260929 16:03:16.045117 14490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:44331 every 8 connection(s)
I20260929 16:03:16.046499 14423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:16.048066 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 14423
I20260929 16:03:16.048247 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:16.048668 24441 external_mini_cluster.cc:1428] Setting key 1e7ecf4453cc7de04772eacc5db2fcf8
I20260929 16:03:16.050918 14491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:16.058183 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:16.050627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43897 (local address 127.23.222.126:44331)
0929 16:03:16.051499 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:16.051515 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:16.051567 (+ 52us) server_negotiation.cc:407] Connection header received
0929 16:03:16.053927 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:16.053955 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:16.054436 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:16.054920 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:16.056117 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:16.056131 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:16.056148 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:16.056174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:16.056264 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:16.056293 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:16.056544 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:16.056680 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 16:03:16.056888 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":348,"threads_started":1}
I20260929 16:03:16.062108 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:16.065147 14491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b: Bootstrap starting.
I20260929 16:03:16.068856 14491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:16.069429 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:16.066737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52249 (local address 127.23.222.126:44331)
0929 16:03:16.066933 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:16.066940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:16.067210 (+ 270us) server_negotiation.cc:407] Connection header received
0929 16:03:16.067427 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:16.067432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:16.067498 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:16.067606 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:16.068651 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:16.068659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:16.068666 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 16:03:16.068687 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:16.068746 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 16:03:16.068769 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:16.068906 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:16.069075 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 16:03:16.069144 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 16:03:16.070194 14491 log.cc:824] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:16.073079 14491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b: No bootstrap required, opened a new log
I20260929 16:03:16.080564 14491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205d935f053142b5b0b3dde49e74960b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44331 } }
I20260929 16:03:16.080989 14491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 16:03:16.081100 14491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205d935f053142b5b0b3dde49e74960b, State: Initialized, Role: FOLLOWER
I20260929 16:03:16.081676 14491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205d935f053142b5b0b3dde49e74960b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44331 } }
I20260929 16:03:16.081924 14491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 16:03:16.082072 14491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 16:03:16.082254 14491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:16.084573 14491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205d935f053142b5b0b3dde49e74960b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44331 } }
I20260929 16:03:16.085114 14491 leader_election.cc:302] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 205d935f053142b5b0b3dde49e74960b; no voters:
I20260929 16:03:16.085749 14491 leader_election.cc:288] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 16:03:16.085996 14501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:16.086881 14501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [term 1 LEADER]: Becoming Leader. State: Replica: 205d935f053142b5b0b3dde49e74960b, State: Running, Role: LEADER
I20260929 16:03:16.087574 14501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205d935f053142b5b0b3dde49e74960b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44331 } }
I20260929 16:03:16.088155 14491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:16.090847 14503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 205d935f053142b5b0b3dde49e74960b. Latest consensus state: current_term: 1 leader_uuid: "205d935f053142b5b0b3dde49e74960b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205d935f053142b5b0b3dde49e74960b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44331 } } }
I20260929 16:03:16.090876 14502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "205d935f053142b5b0b3dde49e74960b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205d935f053142b5b0b3dde49e74960b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44331 } } }
I20260929 16:03:16.091665 14503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:16.091730 14502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:16.100143 14508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:16.106833 14508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:16.117528 14508 catalog_manager.cc:1347] Generated new cluster ID: f88e2ed3796141c89b0935cfcb0473c0
I20260929 16:03:16.117666 14508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:16.143990 14508 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:16.144536 14508 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:16.157723 14508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 205d935f053142b5b0b3dde49e74960b: Generated new TSK 0
I20260929 16:03:16.158955 14508 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:16.170224 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14423
W20260929 16:03:16.171482 14486 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43897
W20260929 16:03:16.172310 24441 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14423: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 16:03:16.172515 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 14423
2026-09-29T16:03:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1956 ms)
[----------] 18 tests from AuthTokenIssuingTest (35983 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:16 dist-test-slave-cq88 krb5kdc[14522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:16 dist-test-slave-cq88 krb5kdc[14522](info): set up 2 sockets
Sep 29 16:03:16 dist-test-slave-cq88 krb5kdc[14522](info): commencing operation
krb5kdc: starting...
W20260929 16:03:17.298969 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:17 dist-test-slave-cq88 krb5kdc[14522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:03:17.575219 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44815
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:44815,127.23.222.125:41543,127.23.222.124:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:17.924925 14535 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:17.925277 14535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:17.925448 14535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:17.925612 14535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:17.933269 14535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:17.933384 14535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:17.933483 14535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:17.933568 14535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:17.933663 14535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:17.944130 14535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:44815,127.23.222.125:41543,127.23.222.124:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:17.945539 14535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:17.946919 14535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:17.959232 14540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:17.959261 14541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:17.959371 14543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:17.960021 14535 server_base.cc:1034] running on GCE node
I20260929 16:03:17.960918 14535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:17.962430 14535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:17.963678 14535 hybrid_clock.cc:648] HybridClock initialized: now 1790697797963602 us; error 30 us; skew 500 ppm
I20260929 16:03:17.964071 14535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:17 dist-test-slave-cq88 krb5kdc[14522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697797, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:17.967401 14535 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:03:17.968940 14535 webserver.cc:460] Webserver started at http://127.23.222.126:37589/ using document root <none> and password file <none>
I20260929 16:03:17.969684 14535 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:17.969820 14535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:17.970156 14535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:17.973268 14535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "ea77c9b237fb4b49a32a43c830f0152b"
format_stamp: "Formatted at 2026-09-29 16:03:17 on dist-test-slave-cq88"
server_key: "3db7b543aff94449943f6163e0a36190"
server_key_iv: "7f206dab6b8d38b776f46bd9d458e887"
server_key_version: "encryptionkey@0"
I20260929 16:03:17.974097 14535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "ea77c9b237fb4b49a32a43c830f0152b"
format_stamp: "Formatted at 2026-09-29 16:03:17 on dist-test-slave-cq88"
server_key: "3db7b543aff94449943f6163e0a36190"
server_key_iv: "7f206dab6b8d38b776f46bd9d458e887"
server_key_version: "encryptionkey@0"
I20260929 16:03:17.979548 14535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:03:17.983222 14550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:17.984673 14535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:17.984884 14535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "ea77c9b237fb4b49a32a43c830f0152b"
format_stamp: "Formatted at 2026-09-29 16:03:17 on dist-test-slave-cq88"
server_key: "3db7b543aff94449943f6163e0a36190"
server_key_iv: "7f206dab6b8d38b776f46bd9d458e887"
server_key_version: "encryptionkey@0"
I20260929 16:03:17.985118 14535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:18.000813 14535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:18.005828 14535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:18.006172 14535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:18.026458 14535 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:44815
I20260929 16:03:18.026508 14602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:44815 every 8 connection(s)
I20260929 16:03:18.028002 14535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:18.032222 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:18.033041 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 14535
I20260929 16:03:18.033336 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:18.034246 24441 external_mini_cluster.cc:1428] Setting key 179d9f6985d36e63be154b49ca894bba
I20260929 16:03:18.043767 14603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:18 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697797, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.222.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.125@KRBTEST.COM" created.
I20260929 16:03:18.070348 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.048017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.23.222.126:44815)
0929 16:03:18.048953 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:18.048972 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:18.049607 (+ 635us) server_negotiation.cc:407] Connection header received
0929 16:03:18.051782 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:18.051793 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:18.052115 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:18.052346 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:18.053543 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.055613 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.057100 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.058148 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.058392 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:18.062075 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:18.062087 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:18.062108 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:18.062147 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:18.065594 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:18.066492 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:18.066499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:18.066501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:18.066670 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:18.067200 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:18.067204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:18.067206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:18.067834 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:18.068012 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:18.068827 (+ 815us) server_negotiation.cc:299] Negotiation successful
0929 16:03:18.069130 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":665,"thread_start_us":414,"threads_started":1}
I20260929 16:03:18.070403 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.048017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:44815 (local address 127.0.0.1:45166)
0929 16:03:18.049381 (+ 1364us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.049435 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:18.051377 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:18.052370 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:18.052374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:18.052964 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:18.053318 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:18.053335 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.055783 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.055787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:18.056932 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:18.056955 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.058400 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.058405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:18.058624 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:18.059426 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:18.059454 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:18.061811 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:18.065745 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:18.065758 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:18.065768 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:18.066357 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:18.066851 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:18.066854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:18.066857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:18.067075 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:18.067995 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:18.068006 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:18.068316 (+ 310us) client_negotiation.cc:769] Sending connection context
0929 16:03:18.068678 (+ 362us) client_negotiation.cc:240] Negotiation successful
0929 16:03:18.069316 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":130,"spinlock_wait_cycles":49408,"thread_start_us":865,"threads_started":1}
I20260929 16:03:18.073988 14603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:18.075297 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.074710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42598)
0929 16:03:18.074933 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.075093 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 16:03:18.075619 14554 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
I20260929 16:03:18.077438 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.076998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42612)
0929 16:03:18.077195 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.077281 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 16:03:18.077621 14603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:41543: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab.
Entry for principal kudu/127.23.222.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab.
I20260929 16:03:18.101203 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.125:41543
--webserver_interface=127.23.222.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab
--principal=kudu/127.23.222.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:44815,127.23.222.125:41543,127.23.222.124:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:03:18.126997 14603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } attempt: 1
I20260929 16:03:18.128362 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.127778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42628)
0929 16:03:18.128033 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.128179 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 16:03:18.129432 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.129033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42636)
0929 16:03:18.129182 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.129268 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:03:18.129730 14603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:41543: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
I20260929 16:03:18.198046 14603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } attempt: 2
I20260929 16:03:18.199404 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.198773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42638)
0929 16:03:18.199070 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.199199 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:03:18.200549 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.200135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42640)
0929 16:03:18.200303 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.200391 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:03:18.200801 14603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:41543: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
I20260929 16:03:18.292075 14603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } attempt: 3
I20260929 16:03:18.293303 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.292786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42656)
0929 16:03:18.293022 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.293134 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 16:03:18.294447 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.294045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42668)
0929 16:03:18.294193 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.294278 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:03:18.294658 14603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:41543: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
I20260929 16:03:18.437914 14603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } attempt: 4
I20260929 16:03:18.439371 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.438658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42682)
0929 16:03:18.438996 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.439107 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260929 16:03:18.440505 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.440077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42688)
0929 16:03:18.440237 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.440329 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 16:03:18.440724 14603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:41543: Network error: Client connection negotiation failed: client connection to 127.23.222.125:41543: connect: Connection refused (error 111)
W20260929 16:03:18.452039 14610 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:18.452405 14610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:18.452543 14610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:18.452669 14610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:18.460393 14610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:18.460537 14610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:18.460640 14610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:18.460765 14610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:18.460887 14610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:18.471729 14610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:44815,127.23.222.125:41543,127.23.222.124:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab
--principal=kudu/127.23.222.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.125:41543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.222.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:18.473088 14610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:18.474566 14610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:18.486681 14615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:18.486733 14618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:18.486764 14616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:18.488055 14610 server_base.cc:1034] running on GCE node
I20260929 16:03:18.489006 14610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:18.490559 14610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:18.491779 14610 hybrid_clock.cc:648] HybridClock initialized: now 1790697798491716 us; error 44 us; skew 500 ppm
I20260929 16:03:18.492139 14610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:18 dist-test-slave-cq88 krb5kdc[14522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697798, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:18.495419 14610 init.cc:377] Logged in from keytab as kudu/127.23.222.125@KRBTEST.COM (short username kudu)
I20260929 16:03:18.496985 14610 webserver.cc:460] Webserver started at http://127.23.222.125:37803/ using document root <none> and password file <none>
I20260929 16:03:18.497692 14610 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:18.497825 14610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:18.498155 14610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:18.501204 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/data/instance:
uuid: "af207581356342bdb08b1efba0997a2f"
format_stamp: "Formatted at 2026-09-29 16:03:18 on dist-test-slave-cq88"
server_key: "7dac61174cd2efa759580dc6475d8d13"
server_key_iv: "77ac5a86fbac4ea1d1af47a80b232fc4"
server_key_version: "encryptionkey@0"
I20260929 16:03:18.502015 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/wal/instance:
uuid: "af207581356342bdb08b1efba0997a2f"
format_stamp: "Formatted at 2026-09-29 16:03:18 on dist-test-slave-cq88"
server_key: "7dac61174cd2efa759580dc6475d8d13"
server_key_iv: "77ac5a86fbac4ea1d1af47a80b232fc4"
server_key_version: "encryptionkey@0"
I20260929 16:03:18.507301 14610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:03:18.510807 14625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:18.512303 14610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 16:03:18.512501 14610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/wal
uuid: "af207581356342bdb08b1efba0997a2f"
format_stamp: "Formatted at 2026-09-29 16:03:18 on dist-test-slave-cq88"
server_key: "7dac61174cd2efa759580dc6475d8d13"
server_key_iv: "77ac5a86fbac4ea1d1af47a80b232fc4"
server_key_version: "encryptionkey@0"
I20260929 16:03:18.512737 14610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:18.533258 14610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:18.538017 14610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:18.538348 14610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:18.558494 14610 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.125:41543
I20260929 16:03:18.558516 14676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.125:41543 every 8 connection(s)
I20260929 16:03:18.560017 14610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
I20260929 16:03:18.563958 14677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:18.569945 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 14610
I20260929 16:03:18.570144 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-1/wal/instance
I20260929 16:03:18.570644 24441 external_mini_cluster.cc:1428] Setting key 57864b3d66f8c58d737227ec6d77a739
I20260929 16:03:18.572057 14677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:18 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697798, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:03:18.597523 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.575790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45176 (local address 127.23.222.126:44815)
0929 16:03:18.576339 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:18.576347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:18.577525 (+ 1178us) server_negotiation.cc:407] Connection header received
0929 16:03:18.580740 (+ 3215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:18.580745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:18.580843 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:18.580962 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:18.582830 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.584118 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.585713 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.586451 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.586603 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:18.590385 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:18.590394 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:18.590402 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:18.590469 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:18.594278 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:18.595084 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:18.595091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:18.595094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:18.595230 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:18.595718 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:18.595723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:18.595727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:18.596023 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:18.596173 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:18.596900 (+ 727us) server_negotiation.cc:299] Negotiation successful
0929 16:03:18.597129 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
I20260929 16:03:18.598513 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.576468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:44815 (local address 127.0.0.1:45176)
0929 16:03:18.577308 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.577371 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:18.580336 (+ 2965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:18.581305 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:18.581315 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:18.581897 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:18.582525 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:18.582545 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.584291 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.584295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:18.585555 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:18.585568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.586682 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.586687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:18.586903 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:18.587664 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:18.587688 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:18.590115 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:18.594438 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:18.594448 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:18.594466 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:18.594915 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:18.595398 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:18.595404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:18.595408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:18.595598 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:18.596163 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:18.596172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:18.596534 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 16:03:18.596813 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 16:03:18.597356 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":305,"threads_started":1}
WARNING: no policy specified for kudu/127.23.222.124@KRBTEST.COM; defaulting to no policy
I20260929 16:03:18.600350 14677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.124@KRBTEST.COM" created.
Sep 29 16:03:18 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697798, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
I20260929 16:03:18.618136 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.601085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42700)
0929 16:03:18.601284 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.601305 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:18.601415 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:18.602507 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:18.602515 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:18.603068 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:18.603346 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:18.603363 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.604797 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.604806 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:18.606352 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:18.606370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.607298 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.607306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:18.607549 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:18.608228 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:18.608255 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:18.611024 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:18.615439 (+ 4415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:18.615444 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:18.615447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:18.615834 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:18.616327 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:18.616332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:18.616334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:18.616498 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:18.617314 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:18.617317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:18.617487 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 16:03:18.617605 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:03:18.617867 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:03:18.618595 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.601413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42700 (local address 127.23.222.125:41543)
0929 16:03:18.602056 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:18.602061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:18.602088 (+ 27us) server_negotiation.cc:407] Connection header received
0929 16:03:18.602132 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:18.602139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:18.602320 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:18.602447 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:18.603571 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.604635 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.606506 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.607084 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.607182 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:18.611284 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:18.611305 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:18.611320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:18.611350 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:18.615284 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:18.615984 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:18.615996 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:18.616002 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:18.616153 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:18.616633 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:18.616640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:18.616646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:18.617176 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:18.617399 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:18.617787 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 16:03:18.618082 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":337,"threads_started":1}
I20260929 16:03:18.621148 14677 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:18.622380 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.621914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57600)
0929 16:03:18.622097 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.622227 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 16:03:18.622716 14628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
I20260929 16:03:18.624469 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.624081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57602)
0929 16:03:18.624262 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.624345 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 16:03:18.624704 14677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:35761: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab.
Entry for principal kudu/127.23.222.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab.
I20260929 16:03:18.639655 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.124:35761
--webserver_interface=127.23.222.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab
--principal=kudu/127.23.222.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:44815,127.23.222.125:41543,127.23.222.124:35761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:03:18.674079 14677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } attempt: 1
I20260929 16:03:18.675424 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.674872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57608)
0929 16:03:18.675102 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.675194 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 16:03:18.676479 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.676106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57620)
0929 16:03:18.676287 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.676367 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:03:18.676739 14677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:35761: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
I20260929 16:03:18.740042 14603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } attempt: 5
I20260929 16:03:18.745010 14677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } attempt: 2
I20260929 16:03:18.746116 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.745599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57624)
0929 16:03:18.745806 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.745916 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:03:18.747318 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.746879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57636)
0929 16:03:18.747044 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.747132 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 16:03:18.747624 14677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:35761: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Sep 29 16:03:18 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697797, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
I20260929 16:03:18.754451 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.740944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42702)
0929 16:03:18.741150 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.741172 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:18.741323 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:18.741773 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:18.741780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:18.742193 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:18.742358 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:18.742366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.743443 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.743451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:18.744370 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:18.744382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.745188 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.745194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:18.745417 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:18.746212 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:18.746244 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:18.748751 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:18.751901 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:18.751908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:18.751912 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:18.752323 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:18.752719 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:18.752724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:18.752727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:18.752881 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:18.753450 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:18.753455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:18.753618 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 16:03:18.753765 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 16:03:18.754107 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 16:03:18.754451 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.741216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42702 (local address 127.23.222.125:41543)
0929 16:03:18.741451 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:18.741458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:18.741481 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:03:18.741533 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:18.741537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:18.741613 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:18.741727 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:18.742539 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.743304 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.744536 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:18.745019 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:18.745169 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:18.748960 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:18.748965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:18.748968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:18.749001 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:18.751747 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:18.752462 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:18.752468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:18.752475 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:18.752600 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:18.752999 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:18.753005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:18.753008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:18.753319 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:18.753486 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:18.753857 (+ 371us) server_negotiation.cc:299] Negotiation successful
0929 16:03:18.754097 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 16:03:18.756086 14603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:18.757333 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.756920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57650)
0929 16:03:18.757099 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.757196 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 16:03:18.758446 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.758021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57660)
0929 16:03:18.758192 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.758282 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:03:18.758701 14603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:35761: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
I20260929 16:03:18.809979 14603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } attempt: 1
I20260929 16:03:18.811404 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.810826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57664)
0929 16:03:18.811053 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.811154 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 16:03:18.812604 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.812224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57668)
0929 16:03:18.812384 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.812488 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 16:03:18.812878 14603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:35761: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
I20260929 16:03:18.838877 14677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } attempt: 3
I20260929 16:03:18.840121 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.839668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57674)
0929 16:03:18.839858 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.839959 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:03:18.841264 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.840835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57682)
0929 16:03:18.841077 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.841153 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260929 16:03:18.841538 14677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:35761: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
I20260929 16:03:18.881131 14603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } attempt: 2
I20260929 16:03:18.882455 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.881940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57686)
0929 16:03:18.882172 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.882270 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 16:03:18.883554 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.883134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57702)
0929 16:03:18.883330 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.883414 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 16:03:18.883762 14603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:35761: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
I20260929 16:03:18.984790 14677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } attempt: 4
I20260929 16:03:18.986032 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.985549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57718)
0929 16:03:18.985773 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.985867 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 16:03:18.986987 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.986666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57722)
0929 16:03:18.986811 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.986886 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 16:03:18.987195 14677 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:35761: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
W20260929 16:03:18.989212 14685 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:18.989580 14685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:18.989702 14685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:18.989866 14685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 16:03:18.989986 14603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } attempt: 3
I20260929 16:03:18.990949 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.990525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57726)
0929 16:03:18.990701 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.990789 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:03:18.992038 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:18.991679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57740)
0929 16:03:18.991857 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:18.991941 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260929 16:03:18.992267 14603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:35761: Network error: Client connection negotiation failed: client connection to 127.23.222.124:35761: connect: Connection refused (error 111)
W20260929 16:03:18.998150 14685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:18.998274 14685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:18.998373 14685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:18.998469 14685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:18.998556 14685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:19.009191 14685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:44815,127.23.222.125:41543,127.23.222.124:35761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab
--principal=kudu/127.23.222.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.124:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.222.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:19.010613 14685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:19.012193 14685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:19.026430 14690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:19.026400 14691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:19.027635 14685 server_base.cc:1034] running on GCE node
W20260929 16:03:19.032060 14693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:19.033135 14685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:19.034876 14685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:19.036185 14685 hybrid_clock.cc:648] HybridClock initialized: now 1790697799036113 us; error 51 us; skew 500 ppm
I20260929 16:03:19.036698 14685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:19 dist-test-slave-cq88 krb5kdc[14522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:19.045892 14685 init.cc:377] Logged in from keytab as kudu/127.23.222.124@KRBTEST.COM (short username kudu)
I20260929 16:03:19.047765 14685 webserver.cc:460] Webserver started at http://127.23.222.124:42939/ using document root <none> and password file <none>
I20260929 16:03:19.048578 14685 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:19.048727 14685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:19.049078 14685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:19.052400 14685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/data/instance:
uuid: "22daee4289624e47832886f7f3a0bf2b"
format_stamp: "Formatted at 2026-09-29 16:03:19 on dist-test-slave-cq88"
server_key: "1db7cae87e92445c756b1fa057e3b300"
server_key_iv: "5b4fa99d8af8e0856deead711a1be1fc"
server_key_version: "encryptionkey@0"
I20260929 16:03:19.053262 14685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/wal/instance:
uuid: "22daee4289624e47832886f7f3a0bf2b"
format_stamp: "Formatted at 2026-09-29 16:03:19 on dist-test-slave-cq88"
server_key: "1db7cae87e92445c756b1fa057e3b300"
server_key_iv: "5b4fa99d8af8e0856deead711a1be1fc"
server_key_version: "encryptionkey@0"
I20260929 16:03:19.058884 14685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 16:03:19.062654 14700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:19.064134 14685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:19.064339 14685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/wal
uuid: "22daee4289624e47832886f7f3a0bf2b"
format_stamp: "Formatted at 2026-09-29 16:03:19 on dist-test-slave-cq88"
server_key: "1db7cae87e92445c756b1fa057e3b300"
server_key_iv: "5b4fa99d8af8e0856deead711a1be1fc"
server_key_version: "encryptionkey@0"
I20260929 16:03:19.064592 14685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:19.084600 14685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:19.089408 14685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:19.089701 14685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:19.109616 14685 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.124:35761
I20260929 16:03:19.109630 14752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.124:35761 every 8 connection(s)
I20260929 16:03:19.111490 14685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
I20260929 16:03:19.115394 14753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:19.116638 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 14685
I20260929 16:03:19.116829 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790697431711191-24441-0/minicluster-data/master-2/wal/instance
I20260929 16:03:19.117379 24441 external_mini_cluster.cc:1428] Setting key 379de0c254b86e765f41358a7dc9992a
I20260929 16:03:19.124663 14753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:19 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
Sep 29 16:03:19 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:03:19.140411 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.119483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.23.222.124:35761)
0929 16:03:19.120357 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.120369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.120412 (+ 43us) server_negotiation.cc:407] Connection header received
0929 16:03:19.122096 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.122119 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.122536 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.122949 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.124513 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.125890 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.127297 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.128388 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.128590 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.133282 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.133289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.133300 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.133337 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.136313 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.137049 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.137056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.137063 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.137208 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.137606 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.137612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.137614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.138127 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.138437 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.138895 (+ 458us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.139383 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":398,"threads_started":1}
I20260929 16:03:19.142817 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.127308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.23.222.126:44815)
0929 16:03:19.127822 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.127828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.128553 (+ 725us) server_negotiation.cc:407] Connection header received
0929 16:03:19.128840 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.128845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.128953 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.129089 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.130259 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.131464 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.133135 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.133558 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.133693 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.137165 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.137170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.137174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.137213 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.140000 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.140773 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.140782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.140785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.140919 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.141299 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.141305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.141308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.141570 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.141726 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.142174 (+ 448us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.142393 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260929 16:03:19.142863 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.127330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:44815 (local address 127.0.0.1:45178)
0929 16:03:19.128418 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:19.128472 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:19.128679 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:19.129181 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:19.129189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:19.129845 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:19.130093 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.130102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.131640 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.131645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.132974 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.132986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.133764 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.133773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.134005 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.134694 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:19.134721 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:19.136972 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:19.140133 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.140145 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.140152 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.140648 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.141044 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.141049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.141052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.141190 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.141698 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:19.141705 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:19.141897 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 16:03:19.142081 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 16:03:19.142392 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":674,"threads_started":1}
I20260929 16:03:19.144616 14753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:19 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
Sep 29 16:03:19 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:03:19.159936 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.145260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:41543 (local address 127.0.0.1:42714)
0929 16:03:19.145604 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:19.145617 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:19.145692 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:19.146027 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:19.146032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:19.146470 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:19.146656 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.146667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.148102 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.148106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.149332 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.149346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.150299 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.150303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.150476 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.151063 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:19.151081 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:19.153279 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:19.156898 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.156906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.156911 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.157478 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.158055 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.158058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.158061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.158207 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.158909 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:19.158913 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:19.159076 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 16:03:19.159194 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 16:03:19.159567 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260929 16:03:19.159960 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.145456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.23.222.125:41543)
0929 16:03:19.145654 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.145661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.145683 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:03:19.145831 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.145836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.145895 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.145976 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.146832 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.147963 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.149476 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.150105 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.150257 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.153512 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.153520 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.153526 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.153578 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.156736 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.157635 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.157640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.157642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.157905 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.158335 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.158343 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.158349 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.158740 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.158953 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.159402 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.159584 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 16:03:19.161005 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.143660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.23.222.126:44815)
0929 16:03:19.143807 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.143812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.143861 (+ 49us) server_negotiation.cc:407] Connection header received
0929 16:03:19.144042 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.144055 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.144127 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.144232 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.145782 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.146872 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.148587 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.149228 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.149356 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.154377 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.154385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.154391 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.154436 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.157656 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.158585 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.158589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.158591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.158723 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.159328 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.159333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.159336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.159704 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.160012 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.160360 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.160612 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 16:03:19.161489 14753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:19.169477 14603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } attempt: 4
Sep 29 16:03:19 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
Sep 29 16:03:19 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
I20260929 16:03:19.178601 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.162336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57762)
0929 16:03:19.162538 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:19.162560 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:19.162657 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:19.163064 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:19.163067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:19.163527 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:19.163661 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.163669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.165048 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.165054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.166380 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.166392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.167468 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.167472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.167713 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.168403 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:19.168428 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:19.171519 (+ 3091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:19.175538 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.175546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.175550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.176191 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.176614 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.176621 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.176627 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.176806 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.177492 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:19.177499 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:19.177671 (+ 172us) client_negotiation.cc:769] Sending connection context
0929 16:03:19.177854 (+ 183us) client_negotiation.cc:240] Negotiation successful
0929 16:03:19.178171 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
Sep 29 16:03:19 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697797, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
I20260929 16:03:19.179541 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.162414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.23.222.124:35761)
0929 16:03:19.162602 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.162607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.162626 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:03:19.162780 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.162785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.162852 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.162919 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.163810 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.164911 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.166551 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.167194 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.167389 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.171760 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.171767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.171774 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.171837 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.175375 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.176335 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.176342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.176346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.176480 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.176956 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.176962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.176966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.177345 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.177497 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.178915 (+ 1418us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.179157 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 16:03:19.181571 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.162756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.23.222.125:41543)
0929 16:03:19.162920 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.162925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.162942 (+ 17us) server_negotiation.cc:407] Connection header received
0929 16:03:19.163122 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.163127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.163194 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.163342 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.164881 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.166055 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.167800 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.168460 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.168601 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.174743 (+ 6142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.174751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.174756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.174807 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.178373 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.179273 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.179280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.179284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.179491 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.179956 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.179962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.179966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.180336 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.180613 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.180954 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.181209 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:03:19.185990 14753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b: Bootstrap starting.
I20260929 16:03:19.187199 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.170166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.23.222.124:35761)
0929 16:03:19.171344 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.171348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.171371 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:03:19.171411 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.171415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.171466 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.171582 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.172423 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.173510 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.175106 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.175854 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.175991 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.180481 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.180489 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.180495 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.180529 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.183846 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.184675 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.184680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.184683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.184866 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.185331 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.185336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.185340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.185647 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.185804 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.186594 (+ 790us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.186799 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":771,"threads_started":1}
I20260929 16:03:19.187856 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.170147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57772)
0929 16:03:19.170501 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:19.170526 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:19.170652 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:19.171632 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:19.171639 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:19.172072 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:19.172242 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.172251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.173646 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.173651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.174914 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.174932 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.176048 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.176059 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.176314 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.177273 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:19.177292 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:19.180216 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:19.183994 (+ 3778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.184000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.184004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.184550 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.185017 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.185022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.185026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.185212 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.185789 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:19.185794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:19.186000 (+ 206us) client_negotiation.cc:769] Sending connection context
0929 16:03:19.186475 (+ 475us) client_negotiation.cc:240] Negotiation successful
0929 16:03:19.187239 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260929 16:03:19.188645 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:19.190125 14753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:19.191193 14753 log.cc:824] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:19.194038 14753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b: No bootstrap required, opened a new log
I20260929 16:03:19.194104 14603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b: Bootstrap starting.
I20260929 16:03:19.192392 14578 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ea77c9b237fb4b49a32a43c830f0152b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45192
I20260929 16:03:19.198225 14603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:19.199533 14603 log.cc:824] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:19.202246 14603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b: No bootstrap required, opened a new log
I20260929 16:03:19.202335 14753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } }
I20260929 16:03:19.202749 14753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22daee4289624e47832886f7f3a0bf2b, State: Initialized, Role: FOLLOWER
I20260929 16:03:19.203274 14753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } }
I20260929 16:03:19.204223 14762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.204639 14762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:19.205371 14753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:19.209841 14603 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } }
I20260929 16:03:19.210284 14603 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea77c9b237fb4b49a32a43c830f0152b, State: Initialized, Role: FOLLOWER
I20260929 16:03:19.210789 14603 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } }
I20260929 16:03:19.211824 14774 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.212194 14774 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:19.212972 14603 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:19.213156 14578 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:03:19.213421 14578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:19.216830 14773 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 16:03:19.217931 14578 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } }
I20260929 16:03:19.219620 14578 leader_election.cc:288] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [CANDIDATE]: Term 1 election: Requested vote from peers af207581356342bdb08b1efba0997a2f (127.23.222.125:41543), 22daee4289624e47832886f7f3a0bf2b (127.23.222.124:35761)
I20260929 16:03:19.220669 14728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea77c9b237fb4b49a32a43c830f0152b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "22daee4289624e47832886f7f3a0bf2b"
I20260929 16:03:19.221187 14728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:19.224953 14728 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea77c9b237fb4b49a32a43c830f0152b in term 1.
I20260929 16:03:19.225723 14553 leader_election.cc:302] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22daee4289624e47832886f7f3a0bf2b, ea77c9b237fb4b49a32a43c830f0152b; no voters:
I20260929 16:03:19.226261 14774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:19.227015 14774 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [term 1 LEADER]: Becoming Leader. State: Replica: ea77c9b237fb4b49a32a43c830f0152b, State: Running, Role: LEADER
I20260929 16:03:19.227624 14774 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } }
I20260929 16:03:19.220811 14685 master_runner.cc:186] Error getting master registration for 127.23.222.126:44815: OK, instance_id { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" instance_seqno: 1790697798020585 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:03:19.231822 14685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:03:19.231372 14784 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea77c9b237fb4b49a32a43c830f0152b. Latest consensus state: current_term: 1 leader_uuid: "ea77c9b237fb4b49a32a43c830f0152b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.231935 14784 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:19.241794 14787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:19.254859 14787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:19.255770 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.231119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.23.222.126:44815)
0929 16:03:19.231332 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.231338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.237371 (+ 6033us) server_negotiation.cc:407] Connection header received
0929 16:03:19.237869 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.237875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.237969 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.238052 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.240557 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.241516 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.243676 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.244494 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.244607 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.248331 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.248339 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.248344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.248406 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.252206 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.253055 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.253060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.253064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.253234 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.253703 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.253709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.253714 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.254130 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.254434 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.255047 (+ 613us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.255340 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:03:19.256711 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.225014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.23.222.124:35761)
0929 16:03:19.225220 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.225229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.233118 (+ 7889us) server_negotiation.cc:407] Connection header received
0929 16:03:19.233581 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.233590 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.233748 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.233883 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.236734 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.237840 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.240251 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.240932 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.241054 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.246124 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.246132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.246140 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.246190 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.249910 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.251928 (+ 2018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.251934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.251937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.252202 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.252999 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.253004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.253007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.253685 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.254067 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.255223 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.255765 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 16:03:19.259924 14556 tablet.cc:2369] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:03:19.263706 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.233326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.23.222.125:41543)
0929 16:03:19.233525 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.233531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.245134 (+ 11603us) server_negotiation.cc:407] Connection header received
0929 16:03:19.245557 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.245565 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.245672 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.245758 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.248071 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.249207 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.251918 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.252649 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.252758 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.256405 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.256413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.256418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.256466 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.260526 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.261428 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.261434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.261438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.261586 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.262006 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.262013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.262016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.262368 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.262625 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.262994 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.263358 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:03:19.275449 14728 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [term 1 FOLLOWER]: Refusing update from remote peer ea77c9b237fb4b49a32a43c830f0152b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:19.276763 14784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [LEADER]: Connected to new peer: Peer: permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:19.266587 14535 master_runner.cc:186] Error getting master registration for 127.23.222.125:41543: OK, instance_id { permanent_uuid: "af207581356342bdb08b1efba0997a2f" instance_seqno: 1790697798552459 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:03:19.281107 14535 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:03:19.286489 14677 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } attempt: 5
Sep 29 16:03:19 dist-test-slave-cq88 krb5kdc[14522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697798, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
I20260929 16:03:19.300849 14762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea77c9b237fb4b49a32a43c830f0152b. Latest consensus state: current_term: 1 leader_uuid: "ea77c9b237fb4b49a32a43c830f0152b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.301157 14762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:19.303680 14774 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea77c9b237fb4b49a32a43c830f0152b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.304018 14774 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:19.305015 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.287204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:35761 (local address 127.0.0.1:57800)
0929 16:03:19.287458 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:19.287479 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:19.287598 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:19.288095 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:19.288103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:19.288671 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:19.288907 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.288919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.290355 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.290360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.291426 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:19.291438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.292423 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.292429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:19.292649 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.293377 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:19.293400 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:19.296408 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:19.300862 (+ 4454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.300869 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.300873 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.301496 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.302012 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:19.302020 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:19.302025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:19.302871 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:19.303782 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:19.303787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:19.304060 (+ 273us) client_negotiation.cc:769] Sending connection context
0929 16:03:19.304270 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 16:03:19.304632 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260929 16:03:19.306480 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.287231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.23.222.124:35761)
0929 16:03:19.287445 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.287451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.287534 (+ 83us) server_negotiation.cc:407] Connection header received
0929 16:03:19.287737 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.287743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.287836 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.287923 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.289077 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.290224 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.291561 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.292259 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.292380 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.296632 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:19.296638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:19.296642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:19.296675 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:19.300673 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.301668 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.301677 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.301681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.301860 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:19.303018 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:19.303023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:19.303026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:19.303539 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:19.303709 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.305522 (+ 1813us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.306087 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 16:03:19.307442 14798 mvcc.cc:205] Tried to move back new op lower bound from 7334698185820475392 to 7334698185641644032. Current Snapshot: MvccSnapshot[applied={T|T < 7334698185820475392}]
I20260929 16:03:19.311527 14787 catalog_manager.cc:1347] Generated new cluster ID: 061d170004184fca9ae48e7b067fa990
I20260929 16:03:19.311625 14787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:19.313263 14762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea77c9b237fb4b49a32a43c830f0152b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.313498 14762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:19.317010 14677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f: Bootstrap starting.
I20260929 16:03:19.325408 14677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:19.326889 14677 log.cc:824] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:19.332837 14677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f: No bootstrap required, opened a new log
I20260929 16:03:19.337687 14787 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:19.339116 14787 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:19.344969 14677 raft_consensus.cc:348] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } }
I20260929 16:03:19.345494 14677 raft_consensus.cc:729] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af207581356342bdb08b1efba0997a2f, State: Initialized, Role: FOLLOWER
I20260929 16:03:19.346093 14677 consensus_queue.cc:260] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } }
I20260929 16:03:19.351830 14803 sys_catalog.cc:455] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.352329 14803 sys_catalog.cc:458] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:19.353248 14677 sys_catalog.cc:564] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:19.353359 14652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea77c9b237fb4b49a32a43c830f0152b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af207581356342bdb08b1efba0997a2f"
I20260929 16:03:19.353932 14652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:19.358632 14652 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea77c9b237fb4b49a32a43c830f0152b in term 1.
I20260929 16:03:19.359726 14651 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [term 1 FOLLOWER]: Refusing update from remote peer ea77c9b237fb4b49a32a43c830f0152b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:19.363528 14784 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [LEADER]: Connected to new peer: Peer: permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:19.364197 14787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b: Generated new TSK 0
I20260929 16:03:19.365150 14787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:19.420868 14803 sys_catalog.cc:455] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea77c9b237fb4b49a32a43c830f0152b. Latest consensus state: current_term: 1 leader_uuid: "ea77c9b237fb4b49a32a43c830f0152b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.421298 14803 sys_catalog.cc:458] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:19.420876 14812 sys_catalog.cc:455] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea77c9b237fb4b49a32a43c830f0152b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.421409 14812 sys_catalog.cc:458] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:19.442884 14820 catalog_manager.cc:1259] Loaded cluster ID: 061d170004184fca9ae48e7b067fa990
I20260929 16:03:19.443023 14820 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f: loading cluster ID for follower catalog manager: success
I20260929 16:03:19.447002 14820 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f: acquiring CA information for follower catalog manager: success
I20260929 16:03:19.449681 14820 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af207581356342bdb08b1efba0997a2f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 16:03:19.480576 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.477864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.23.222.124:35761)
0929 16:03:19.478078 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.478085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.478645 (+ 560us) server_negotiation.cc:407] Connection header received
0929 16:03:19.478944 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.478951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.479039 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.479117 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.480320 (+ 1203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57804: BlockingRecv error: recv got EOF from 127.0.0.1:57804 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 16:03:19.482825 14757 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.478770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.23.222.126:44815)
0929 16:03:19.478994 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.479000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.481200 (+ 2200us) server_negotiation.cc:407] Connection header received
0929 16:03:19.481466 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.481472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.481549 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.481622 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.482561 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45214: BlockingRecv error: recv got EOF from 127.0.0.1:45214 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 16:03:19.486433 14682 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.484334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42734 (local address 127.23.222.125:41543)
0929 16:03:19.484547 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.484553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.484620 (+ 67us) server_negotiation.cc:407] Connection header received
0929 16:03:19.484831 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.484837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.484943 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.485063 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.486165 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42734: BlockingRecv error: recv got EOF from 127.0.0.1:42734 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 16:03:19.499567 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.490424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.23.222.125:41543)
0929 16:03:19.490592 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.490598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.491977 (+ 1379us) server_negotiation.cc:407] Connection header received
0929 16:03:19.492189 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.492195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.492417 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.492591 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:19.493054 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.494215 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.496245 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.496954 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.497052 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.497766 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:19.498466 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:19.498583 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.498820 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.498928 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 16:03:19.515733 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.513338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.23.222.124:35761)
0929 16:03:19.513545 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.513551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.514113 (+ 562us) server_negotiation.cc:407] Connection header received
0929 16:03:19.514386 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.514394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.514535 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.514651 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.515492 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57808: BlockingRecv error: recv got EOF from 127.0.0.1:57808 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:03:19.530419 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.520686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.23.222.125:41543)
0929 16:03:19.520880 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.520887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.523816 (+ 2929us) server_negotiation.cc:407] Connection header received
0929 16:03:19.524036 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.524043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.524175 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.524284 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:19.524749 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.525956 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.528057 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.528744 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.528876 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.529487 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:19.529650 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:19.529717 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.529973 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.530063 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 16:03:19.545521 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.543317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.23.222.124:35761)
0929 16:03:19.543514 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.543521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.543911 (+ 390us) server_negotiation.cc:407] Connection header received
0929 16:03:19.544137 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.544143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.544242 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.544359 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.545264 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57822: BlockingRecv error: recv got EOF from 127.0.0.1:57822 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 16:03:19.590807 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.559413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.23.222.125:41543)
0929 16:03:19.559682 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.559692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.583876 (+ 24184us) server_negotiation.cc:407] Connection header received
0929 16:03:19.584231 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.584239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.584333 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.584400 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:19.585166 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.586465 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.588437 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.589119 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.589222 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.589855 (+ 633us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:19.590066 (+ 211us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:19.590127 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.590356 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.590447 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
W20260929 16:03:19.613341 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.605623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.23.222.124:35761)
0929 16:03:19.605816 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.605822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.611664 (+ 5842us) server_negotiation.cc:407] Connection header received
0929 16:03:19.611874 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.611880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.611984 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.612111 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.613092 (+ 981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57830: BlockingRecv error: recv got EOF from 127.0.0.1:57830 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:03:19.629577 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.622053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.23.222.125:41543)
0929 16:03:19.622306 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.622313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.622892 (+ 579us) server_negotiation.cc:407] Connection header received
0929 16:03:19.623102 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.623108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.623216 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.623382 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:19.623892 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.625213 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.627284 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.628078 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.628215 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.628749 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:19.628898 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:19.628997 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.629163 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.629256 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260929 16:03:19.639396 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.637389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.23.222.124:35761)
0929 16:03:19.637685 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.637693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.637717 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:03:19.638059 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.638065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.638166 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.638288 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.639129 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57842: BlockingRecv error: recv got EOF from 127.0.0.1:57842 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
I20260929 16:03:19.658347 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.650844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.23.222.125:41543)
0929 16:03:19.651046 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.651053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.651638 (+ 585us) server_negotiation.cc:407] Connection header received
0929 16:03:19.651815 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.651822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.651922 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.652045 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:19.652515 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.653820 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.656029 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.656792 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.656939 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.657449 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:19.657640 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:19.657746 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.657922 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.658013 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 16:03:19.668227 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.665654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.23.222.124:35761)
0929 16:03:19.665848 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.665855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.666470 (+ 615us) server_negotiation.cc:407] Connection header received
0929 16:03:19.666687 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.666694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.666793 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.666926 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.667956 (+ 1030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57852: BlockingRecv error: recv got EOF from 127.0.0.1:57852 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 16:03:19.696575 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.688304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.23.222.125:41543)
0929 16:03:19.688828 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.688837 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.689500 (+ 663us) server_negotiation.cc:407] Connection header received
0929 16:03:19.689717 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.689723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.689926 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.690169 (+ 243us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:19.690782 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.692112 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.694202 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.694907 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.695066 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.695668 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:19.695856 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:19.695956 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.696139 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.696243 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348}
W20260929 16:03:19.707782 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.705799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.23.222.124:35761)
0929 16:03:19.706016 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.706023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.706047 (+ 24us) server_negotiation.cc:407] Connection header received
0929 16:03:19.706274 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.706281 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.706377 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.706458 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.707549 (+ 1091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57862: BlockingRecv error: recv got EOF from 127.0.0.1:57862 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 16:03:19.752322 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.744623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.23.222.125:41543)
0929 16:03:19.744873 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.744879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.745474 (+ 595us) server_negotiation.cc:407] Connection header received
0929 16:03:19.745746 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.745754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.745883 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.745993 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:19.746564 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.747804 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.749881 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.750676 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.750813 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.751408 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:19.751570 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:19.751675 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.751870 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.751963 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 16:03:19.762365 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.759918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.23.222.124:35761)
0929 16:03:19.760114 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.760122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.760719 (+ 597us) server_negotiation.cc:407] Connection header received
0929 16:03:19.760941 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.760948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.761066 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.761191 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.762094 (+ 903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57878: BlockingRecv error: recv got EOF from 127.0.0.1:57878 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 16:03:19.837040 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.830469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.23.222.125:41543)
0929 16:03:19.830731 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.830738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.831111 (+ 373us) server_negotiation.cc:407] Connection header received
0929 16:03:19.831407 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.831411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.831488 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.831576 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:19.832111 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.832895 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.834989 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.835543 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.835655 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.836204 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:19.836337 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:19.836415 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.836616 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.836689 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 16:03:19.846705 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.844496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.23.222.124:35761)
0929 16:03:19.844708 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.844715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.845217 (+ 502us) server_negotiation.cc:407] Connection header received
0929 16:03:19.845450 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.845455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.845570 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.845700 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.846449 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57892: BlockingRecv error: recv got EOF from 127.0.0.1:57892 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 16:03:19.970763 14784 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea77c9b237fb4b49a32a43c830f0152b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea77c9b237fb4b49a32a43c830f0152b" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 44815 } } peers { permanent_uuid: "af207581356342bdb08b1efba0997a2f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 41543 } } peers { permanent_uuid: "22daee4289624e47832886f7f3a0bf2b" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 35761 } } }
I20260929 16:03:19.971179 14784 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea77c9b237fb4b49a32a43c830f0152b [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:19.983623 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:19.978192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42796 (local address 127.23.222.125:41543)
0929 16:03:19.978463 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.978471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.978767 (+ 296us) server_negotiation.cc:407] Connection header received
0929 16:03:19.979007 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.979011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.979081 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.979186 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:19.979697 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.980402 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.981757 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:19.982209 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:19.982354 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:19.982772 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:19.982891 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:19.982995 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:19.983220 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 16:03:19.983360 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 16:03:19.991204 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:19.989381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.23.222.124:35761)
0929 16:03:19.989657 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:19.989663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:19.989890 (+ 227us) server_negotiation.cc:407] Connection header received
0929 16:03:19.990057 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:19.990061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:19.990135 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:19.990221 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:19.990984 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57894: BlockingRecv error: recv got EOF from 127.0.0.1:57894 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260929 16:03:20.221432 14773 catalog_manager.cc:1259] Loaded cluster ID: 061d170004184fca9ae48e7b067fa990
I20260929 16:03:20.221622 14773 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b: loading cluster ID for follower catalog manager: success
I20260929 16:03:20.224426 14773 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b: acquiring CA information for follower catalog manager: success
I20260929 16:03:20.226410 14773 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22daee4289624e47832886f7f3a0bf2b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:03:20.261391 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:20.255222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42800 (local address 127.23.222.125:41543)
0929 16:03:20.255463 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:20.255470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:20.255810 (+ 340us) server_negotiation.cc:407] Connection header received
0929 16:03:20.256042 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:20.256048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:20.256135 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:20.256214 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:20.256795 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.257511 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.259588 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.259992 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.260088 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:20.260604 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:20.260710 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:20.260774 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:20.260995 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 16:03:20.261072 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 16:03:20.278115 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:20.270753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.23.222.124:35761)
0929 16:03:20.271015 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:20.271022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:20.271520 (+ 498us) server_negotiation.cc:407] Connection header received
0929 16:03:20.271710 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:20.271717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:20.271947 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:20.272085 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:20.272525 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.273713 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.275142 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.275873 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.276012 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:20.276594 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:20.277366 (+ 772us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:20.277523 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:20.277649 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 16:03:20.277746 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 16:03:20.294375 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:20.285331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57912 (local address 127.23.222.124:35761)
0929 16:03:20.285541 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:20.285548 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:20.286781 (+ 1233us) server_negotiation.cc:407] Connection header received
0929 16:03:20.287088 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:20.287095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:20.287184 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:20.287274 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:20.288210 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.289466 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.291845 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.292581 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.292683 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:20.293461 (+ 778us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:20.293625 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:20.293687 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:20.293941 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 16:03:20.294030 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 16:03:20.299898 14938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:20.285342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.23.222.126:44815)
0929 16:03:20.285953 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:20.285960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:20.291637 (+ 5677us) server_negotiation.cc:407] Connection header received
0929 16:03:20.292251 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:20.292260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:20.292554 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:20.292753 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:20.294653 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.295687 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.297483 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.297922 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.298063 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:20.298639 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:20.299111 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:20.299284 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:20.299429 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 16:03:20.299522 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":337,"threads_started":1}
I20260929 16:03:20.301644 14682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:20.286984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.23.222.125:41543)
0929 16:03:20.287213 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:20.287222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:20.295993 (+ 8771us) server_negotiation.cc:407] Connection header received
0929 16:03:20.296346 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:20.296353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:20.296450 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:20.296543 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:20.296951 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.298059 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.299619 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:20.300157 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:20.300259 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:20.300831 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:20.300960 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:20.301051 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:20.301268 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 16:03:20.301368 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 16:03:20.305141 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14535
I20260929 16:03:20.306478 14598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:20.424458 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 14535
I20260929 16:03:20.455085 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14610
I20260929 16:03:20.456490 14672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:20.565223 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 14610
I20260929 16:03:20.586193 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14685
I20260929 16:03:20.587590 14748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:20.703636 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 14685
2026-09-29T16:03:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4534 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 16:03:20 dist-test-slave-cq88 krb5kdc[14947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 16:03:20 dist-test-slave-cq88 krb5kdc[14947](info): set up 2 sockets
Sep 29 16:03:20 dist-test-slave-cq88 krb5kdc[14947](info): commencing operation
krb5kdc: starting...
W20260929 16:03:21.831043 24441 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/test-user.keytab.
Sep 29 16:03:22 dist-test-slave-cq88 krb5kdc[14947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T16:03:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T16:03:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.222.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
Entry for principal kudu/127.23.222.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab.
I20260929 16:03:22.108971 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:45811
--webserver_interface=127.23.222.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:45811,127.23.222.125:40069,127.23.222.124:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 16:03:22.453801 14960 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:22.454170 14960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:22.454272 14960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:22.454434 14960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:22.461867 14960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:22.461973 14960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:22.462069 14960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:22.462175 14960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:22.462260 14960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:22.472394 14960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:45811,127.23.222.125:40069,127.23.222.124:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.126.keytab
--principal=kudu/127.23.222.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.126:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.222.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:22.473999 14960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:22.475442 14960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:22.487195 14965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:22.487088 14968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:22.487694 14966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:22.488179 14960 server_base.cc:1034] running on GCE node
I20260929 16:03:22.489094 14960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:22.490552 14960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:22.491818 14960 hybrid_clock.cc:648] HybridClock initialized: now 1790697802491770 us; error 56 us; skew 500 ppm
I20260929 16:03:22.492182 14960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:22 dist-test-slave-cq88 krb5kdc[14947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697802, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:22.495674 14960 init.cc:377] Logged in from keytab as kudu/127.23.222.126@KRBTEST.COM (short username kudu)
I20260929 16:03:22.497205 14960 webserver.cc:460] Webserver started at http://127.23.222.126:39187/ using document root <none> and password file <none>
I20260929 16:03:22.497933 14960 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:22.498067 14960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:22.498404 14960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:22.501489 14960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/data/instance:
uuid: "876725c138094625b20077169557e52e"
format_stamp: "Formatted at 2026-09-29 16:03:22 on dist-test-slave-cq88"
server_key: "129ad97af68f8f66a25463c6c20382e4"
server_key_iv: "f244e4387ed0a52fa6904bbfb32e0a20"
server_key_version: "encryptionkey@0"
I20260929 16:03:22.502290 14960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/wal/instance:
uuid: "876725c138094625b20077169557e52e"
format_stamp: "Formatted at 2026-09-29 16:03:22 on dist-test-slave-cq88"
server_key: "129ad97af68f8f66a25463c6c20382e4"
server_key_iv: "f244e4387ed0a52fa6904bbfb32e0a20"
server_key_version: "encryptionkey@0"
I20260929 16:03:22.507637 14960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 16:03:22.510989 14975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:22.512311 14960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 16:03:22.512511 14960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/wal
uuid: "876725c138094625b20077169557e52e"
format_stamp: "Formatted at 2026-09-29 16:03:22 on dist-test-slave-cq88"
server_key: "129ad97af68f8f66a25463c6c20382e4"
server_key_iv: "f244e4387ed0a52fa6904bbfb32e0a20"
server_key_version: "encryptionkey@0"
I20260929 16:03:22.512732 14960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:22.533202 14960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:22.537804 14960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:22.538097 14960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:22.557224 14960 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.126:45811
I20260929 16:03:22.557245 15027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.126:45811 every 8 connection(s)
I20260929 16:03:22.558728 14960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/data/info.pb
I20260929 16:03:22.562907 15028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:22.566797 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 14960
I20260929 16:03:22.567032 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-0/wal/instance
I20260929 16:03:22.567544 24441 external_mini_cluster.cc:1428] Setting key 38b0f350dca5a54c887e49ece829a8ce
I20260929 16:03:22.570715 15028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:22 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697802, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.222.125@KRBTEST.COM; defaulting to no policy
I20260929 16:03:22.596712 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.574299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:45811 (local address 127.0.0.1:38648)
0929 16:03:22.575572 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.575608 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:22.577443 (+ 1835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:22.578412 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:22.578418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:22.579013 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:22.579448 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:22.579488 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:22.581877 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:22.581882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:22.583008 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:22.583021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:22.584457 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:22.584463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:22.584591 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:22.585285 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:22.585309 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:22.587795 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:22.592443 (+ 4648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:22.592452 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:22.592462 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:22.592860 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:22.593361 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:22.593364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:22.593366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:22.593507 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:22.594538 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:22.594543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:22.594725 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 16:03:22.594930 (+ 205us) client_negotiation.cc:240] Negotiation successful
0929 16:03:22.595413 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"spinlock_wait_cycles":47488,"thread_start_us":834,"threads_started":1}
I20260929 16:03:22.596724 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.574293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.23.222.126:45811)
0929 16:03:22.575213 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:22.575231 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:22.575734 (+ 503us) server_negotiation.cc:407] Connection header received
0929 16:03:22.577827 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:22.577838 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:22.578153 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:22.578372 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:22.579689 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:22.581675 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:22.583167 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:22.584187 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:22.584399 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:22.588055 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:22.588069 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:22.588089 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:22.588133 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:22.592291 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:22.593017 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:22.593027 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:22.593032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:22.593242 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:22.593633 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:22.593640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:22.593644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:22.594387 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:22.594616 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:22.595157 (+ 541us) server_negotiation.cc:299] Negotiation successful
0929 16:03:22.595625 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":660,"thread_start_us":422,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.125@KRBTEST.COM" created.
I20260929 16:03:22.600157 15028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:22.601518 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.600946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39898)
0929 16:03:22.601183 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.601337 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 16:03:22.601855 14976 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
I20260929 16:03:22.603770 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.603197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39904)
0929 16:03:22.603474 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.603576 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260929 16:03:22.604027 15028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:40069: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab.
Entry for principal kudu/127.23.222.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab.
I20260929 16:03:22.631703 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.125:40069
--webserver_interface=127.23.222.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab
--principal=kudu/127.23.222.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:45811,127.23.222.125:40069,127.23.222.124:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:03:22.653383 15028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } attempt: 1
I20260929 16:03:22.654506 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.654064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39910)
0929 16:03:22.654256 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.654365 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 16:03:22.655517 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.655135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39924)
0929 16:03:22.655331 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.655394 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 16:03:22.655687 15028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:40069: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
I20260929 16:03:22.723941 15028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } attempt: 2
I20260929 16:03:22.725154 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.724634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39928)
0929 16:03:22.724906 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.725010 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 16:03:22.726140 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.725783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39930)
0929 16:03:22.725929 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.726021 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:03:22.726320 15028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:40069: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
I20260929 16:03:22.817559 15028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } attempt: 3
I20260929 16:03:22.818650 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.818220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39934)
0929 16:03:22.818422 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.818520 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 16:03:22.819681 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.819338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39936)
0929 16:03:22.819474 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.819549 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 16:03:22.819855 15028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:40069: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
I20260929 16:03:22.963125 15028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } attempt: 4
I20260929 16:03:22.964354 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.963821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39938)
0929 16:03:22.964139 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.964224 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260929 16:03:22.965013 15035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260929 16:03:22.965332 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:22.964970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39952)
0929 16:03:22.965108 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:22.965191 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:03:22.965373 15035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:22.965534 15035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:22.965523 15028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.125:40069: Network error: Client connection negotiation failed: client connection to 127.23.222.125:40069: connect: Connection refused (error 111)
W20260929 16:03:22.965677 15035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:22.973183 15035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:22.973315 15035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:22.973441 15035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:22.973531 15035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:22.973640 15035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:22.983846 15035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:45811,127.23.222.125:40069,127.23.222.124:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.125.keytab
--principal=kudu/127.23.222.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.125:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.222.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:22.985205 15035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:22.986552 15035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:22.998409 15040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:22.998369 15041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:22.998442 15043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:22.999266 15035 server_base.cc:1034] running on GCE node
I20260929 16:03:22.999969 15035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:23.001322 15035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:23.002566 15035 hybrid_clock.cc:648] HybridClock initialized: now 1790697803002511 us; error 33 us; skew 500 ppm
I20260929 16:03:23.002892 15035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697803, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:23.006058 15035 init.cc:377] Logged in from keytab as kudu/127.23.222.125@KRBTEST.COM (short username kudu)
I20260929 16:03:23.007598 15035 webserver.cc:460] Webserver started at http://127.23.222.125:45445/ using document root <none> and password file <none>
I20260929 16:03:23.008250 15035 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:23.008333 15035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:23.008587 15035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:23.011687 15035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/data/instance:
uuid: "18a484f6ac6f41edb3fdad7488dd855f"
format_stamp: "Formatted at 2026-09-29 16:03:23 on dist-test-slave-cq88"
server_key: "0d751d377d8a3c5ecd5cd5d6ea9094f2"
server_key_iv: "41502ce18541a67c64d2916cb6ee0a3c"
server_key_version: "encryptionkey@0"
I20260929 16:03:23.012413 15035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/wal/instance:
uuid: "18a484f6ac6f41edb3fdad7488dd855f"
format_stamp: "Formatted at 2026-09-29 16:03:23 on dist-test-slave-cq88"
server_key: "0d751d377d8a3c5ecd5cd5d6ea9094f2"
server_key_iv: "41502ce18541a67c64d2916cb6ee0a3c"
server_key_version: "encryptionkey@0"
I20260929 16:03:23.017767 15035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 16:03:23.021339 15050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:23.022725 15035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 16:03:23.022902 15035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/wal
uuid: "18a484f6ac6f41edb3fdad7488dd855f"
format_stamp: "Formatted at 2026-09-29 16:03:23 on dist-test-slave-cq88"
server_key: "0d751d377d8a3c5ecd5cd5d6ea9094f2"
server_key_iv: "41502ce18541a67c64d2916cb6ee0a3c"
server_key_version: "encryptionkey@0"
I20260929 16:03:23.023113 15035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:23.039759 15035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:23.044256 15035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:23.044550 15035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:23.064354 15035 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.125:40069
I20260929 16:03:23.064378 15102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.125:40069 every 8 connection(s)
I20260929 16:03:23.065742 15035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/data/info.pb
I20260929 16:03:23.068183 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 15035
I20260929 16:03:23.068375 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-1/wal/instance
I20260929 16:03:23.068917 24441 external_mini_cluster.cc:1428] Setting key 275f371d57a01674e776fffcc0babed8
I20260929 16:03:23.070114 15103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:23.080286 15103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697803, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.222.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.222.124@KRBTEST.COM" created.
I20260929 16:03:23.102074 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.083146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38660 (local address 127.23.222.126:45811)
0929 16:03:23.083350 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.083354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.084823 (+ 1469us) server_negotiation.cc:407] Connection header received
0929 16:03:23.086817 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.086824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.086927 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.087055 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.088704 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.089390 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.091000 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.091493 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.091638 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.095610 (+ 3972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.095616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.095619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.095653 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.098420 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.099298 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.099302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.099305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.099485 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.099976 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.099984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.099990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.100405 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.100577 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.101338 (+ 761us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.101636 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 16:03:23.102747 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.083825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:45811 (local address 127.0.0.1:38660)
0929 16:03:23.084643 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.084688 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:23.086521 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:23.087357 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:23.087366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:23.087927 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:23.088518 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.088536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.089578 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.089582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.090813 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.090826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.091773 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.091781 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.092035 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.092788 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:23.092811 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:23.095350 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:23.098581 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.098591 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.098611 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.099130 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.099623 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.099627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.099629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.099829 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.100553 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:23.100560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:23.100948 (+ 388us) client_negotiation.cc:769] Sending connection context
0929 16:03:23.101180 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 16:03:23.101589 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":355,"threads_started":1}
I20260929 16:03:23.104713 15103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697803, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
I20260929 16:03:23.120524 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.105550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39958)
0929 16:03:23.105734 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.105754 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:23.105853 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:23.107092 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:23.107098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:23.107677 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:23.107841 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.107852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.109469 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.109478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.110503 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.110517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.111556 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.111563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.111778 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.112554 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:23.112571 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:23.114800 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:23.118112 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.118120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.118125 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.118478 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.118851 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.118858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.118863 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.118995 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.119672 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:23.119676 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:23.119815 (+ 139us) client_negotiation.cc:769] Sending connection context
0929 16:03:23.119987 (+ 172us) client_negotiation.cc:240] Negotiation successful
0929 16:03:23.120261 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 16:03:23.121026 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.105827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.23.222.125:40069)
0929 16:03:23.106599 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.106606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.106640 (+ 34us) server_negotiation.cc:407] Connection header received
0929 16:03:23.106696 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.106703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.106901 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.107110 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.108051 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.109307 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.110672 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.111388 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.111549 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.115020 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.115033 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.115047 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.115088 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.117973 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.118614 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.118624 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.118627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.118743 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.119103 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.119109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.119113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.119548 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.119766 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.120221 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.120530 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":409,"threads_started":1}
I20260929 16:03:23.123842 15103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:23.125203 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.124592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50542)
0929 16:03:23.124808 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.124973 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 16:03:23.125566 15051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
I20260929 16:03:23.127579 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.127053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50552)
0929 16:03:23.127331 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.127450 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 16:03:23.127820 15103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:36451: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.222.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab.
Entry for principal kudu/127.23.222.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab.
I20260929 16:03:23.134809 24441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGayxzf/build/asan/bin/kudu
/tmp/dist-test-taskGayxzf/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.124:36451
--webserver_interface=127.23.222.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab
--principal=kudu/127.23.222.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.222.84:45773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.222.126:45811,127.23.222.125:40069,127.23.222.124:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 16:03:23.177224 15103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } attempt: 1
I20260929 16:03:23.178568 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.178065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50560)
0929 16:03:23.178286 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.178400 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 16:03:23.179661 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.179232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50562)
0929 16:03:23.179474 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.179551 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 16:03:23.179868 15103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:36451: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
I20260929 16:03:23.248095 15103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } attempt: 2
I20260929 16:03:23.249207 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.248757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50578)
0929 16:03:23.248960 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.249057 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 16:03:23.250223 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.249886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50586)
0929 16:03:23.250041 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.250114 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 16:03:23.250442 15103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:36451: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
I20260929 16:03:23.264750 15028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } attempt: 5
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697802, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
I20260929 16:03:23.280118 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.266054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.23.222.125:40069)
0929 16:03:23.266230 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.266235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.266254 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:03:23.266311 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.266316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.266396 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.266479 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.267512 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.268239 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.269956 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.270563 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.270700 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.274641 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.274646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.274649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.274684 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.277402 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.278082 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.278090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.278093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.278222 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.278608 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.278614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.278617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.278911 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.279069 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.279547 (+ 478us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.279775 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:03:23.280121 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.265728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39968)
0929 16:03:23.265992 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.266017 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:23.266172 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:23.266577 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:23.266582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:23.267067 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:23.267340 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.267353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.268389 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.268394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.269756 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.269772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.270779 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.270788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.271033 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.271721 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:23.271745 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:23.274419 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:23.277558 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.277563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.277568 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.277957 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.278338 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.278344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.278347 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.278494 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.279033 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:23.279038 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:23.279203 (+ 165us) client_negotiation.cc:769] Sending connection context
0929 16:03:23.279436 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 16:03:23.279767 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 16:03:23.281247 15028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:23.282254 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.281872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50602)
0929 16:03:23.282038 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.282121 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:03:23.283191 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.282881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50614)
0929 16:03:23.283021 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.283096 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 16:03:23.283448 15028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:36451: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
I20260929 16:03:23.334672 15028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } attempt: 1
I20260929 16:03:23.335958 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.335487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50624)
0929 16:03:23.335672 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.335774 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 16:03:23.336951 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.336616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50630)
0929 16:03:23.336764 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.336845 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 16:03:23.337162 15028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:36451: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
I20260929 16:03:23.341646 15103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } attempt: 3
I20260929 16:03:23.342646 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.342218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50642)
0929 16:03:23.342400 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.342502 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 16:03:23.343783 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.343356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50656)
0929 16:03:23.343598 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.343671 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260929 16:03:23.343976 15103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:36451: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
I20260929 16:03:23.405376 15028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } attempt: 2
I20260929 16:03:23.406584 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.406082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50658)
0929 16:03:23.406317 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.406408 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 16:03:23.407711 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.407361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50662)
0929 16:03:23.407527 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.407593 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 16:03:23.407897 15028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:36451: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
W20260929 16:03:23.470801 15110 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 16:03:23.471275 15110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 16:03:23.471397 15110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 16:03:23.471557 15110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 16:03:23.478807 15110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 16:03:23.478940 15110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 16:03:23.479050 15110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 16:03:23.479142 15110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 16:03:23.479260 15110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 16:03:23.487171 15103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } attempt: 4
I20260929 16:03:23.488411 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.487845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50674)
0929 16:03:23.488114 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.488220 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 16:03:23.489411 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.489063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50682)
0929 16:03:23.489208 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.489289 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 16:03:23.489627 15103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:36451: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
I20260929 16:03:23.489982 15110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.222.84:45773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.222.126:45811,127.23.222.125:40069,127.23.222.124:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/krb5kdc/kudu_127.23.222.124.keytab
--principal=kudu/127.23.222.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.222.124:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.222.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 13:57:48 UTC on ccycloud.quasar-bocjyr.root.comops.site
build id 4680287
ASAN enabled
I20260929 16:03:23.491328 15110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 16:03:23.492611 15110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 16:03:23.504552 15115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:23.504534 15116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 16:03:23.504587 15118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 16:03:23.505477 15110 server_base.cc:1034] running on GCE node
I20260929 16:03:23.506229 15110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 16:03:23.507647 15110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 16:03:23.508904 15110 hybrid_clock.cc:648] HybridClock initialized: now 1790697803508839 us; error 32 us; skew 500 ppm
I20260929 16:03:23.509302 15110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697803, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 16:03:23.512540 15110 init.cc:377] Logged in from keytab as kudu/127.23.222.124@KRBTEST.COM (short username kudu)
I20260929 16:03:23.514320 15110 webserver.cc:460] Webserver started at http://127.23.222.124:44871/ using document root <none> and password file <none>
I20260929 16:03:23.514132 15028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } attempt: 3
I20260929 16:03:23.515075 15110 fs_manager.cc:359] Metadata directory not provided
I20260929 16:03:23.515211 15110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 16:03:23.515537 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.514923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50694)
0929 16:03:23.515162 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.515333 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 16:03:23.515568 15110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 16:03:23.516748 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.516323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50704)
0929 16:03:23.516543 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.516637 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260929 16:03:23.517016 15028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.222.124:36451: Network error: Client connection negotiation failed: client connection to 127.23.222.124:36451: connect: Connection refused (error 111)
I20260929 16:03:23.518889 15110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/data/instance:
uuid: "dc79b753764943c79d606a9ceeae06bc"
format_stamp: "Formatted at 2026-09-29 16:03:23 on dist-test-slave-cq88"
server_key: "ebdc0661188cfeb0cb690c170e2593c7"
server_key_iv: "97654968ed0c239ac528e1cb1456f288"
server_key_version: "encryptionkey@0"
I20260929 16:03:23.519742 15110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/wal/instance:
uuid: "dc79b753764943c79d606a9ceeae06bc"
format_stamp: "Formatted at 2026-09-29 16:03:23 on dist-test-slave-cq88"
server_key: "ebdc0661188cfeb0cb690c170e2593c7"
server_key_iv: "97654968ed0c239ac528e1cb1456f288"
server_key_version: "encryptionkey@0"
I20260929 16:03:23.525020 15110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 16:03:23.528717 15125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 16:03:23.530176 15110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 16:03:23.530364 15110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/data,/tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/wal
uuid: "dc79b753764943c79d606a9ceeae06bc"
format_stamp: "Formatted at 2026-09-29 16:03:23 on dist-test-slave-cq88"
server_key: "ebdc0661188cfeb0cb690c170e2593c7"
server_key_iv: "97654968ed0c239ac528e1cb1456f288"
server_key_version: "encryptionkey@0"
I20260929 16:03:23.530612 15110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 16:03:23.558537 15110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 16:03:23.562906 15110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 16:03:23.563194 15110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 16:03:23.582280 15110 rpc_server.cc:307] RPC server started. Bound to: 127.23.222.124:36451
I20260929 16:03:23.582312 15176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.222.124:36451 every 8 connection(s)
I20260929 16:03:23.583706 15110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/data/info.pb
I20260929 16:03:23.587435 15177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 16:03:23.590453 24441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGayxzf/build/asan/bin/kudu as pid 15110
I20260929 16:03:23.590646 24441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGayxzf/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790697431711191-24441-0/minicluster-data/master-2/wal/instance
I20260929 16:03:23.591115 24441 external_mini_cluster.cc:1428] Setting key c1f62c4b32a6d49ae143263d240fb9ed
I20260929 16:03:23.595660 15177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697803, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:03:23.615573 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.598585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.23.222.126:45811)
0929 16:03:23.598757 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.598761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.599598 (+ 837us) server_negotiation.cc:407] Connection header received
0929 16:03:23.599813 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.599817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.599884 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.599999 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.601448 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.602347 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.604279 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.604725 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.604818 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.608699 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.608704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.608707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.608738 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.611847 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.612790 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.612795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.612798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.612970 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.613457 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.613462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.613464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.613812 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.613961 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.614718 (+ 757us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.614959 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 16:03:23.616422 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.592989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50718 (local address 127.23.222.124:36451)
0929 16:03:23.593838 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.593850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.593894 (+ 44us) server_negotiation.cc:407] Connection header received
0929 16:03:23.595608 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.595631 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.596068 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.596401 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.598222 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.600177 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.601885 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.602820 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.602942 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.607759 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.607770 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.607789 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.607828 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.612045 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.612821 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.612828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.612831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.613000 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.613425 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.613429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.613432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.614109 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.614332 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.614762 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.615079 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":424,"threads_started":1}
I20260929 16:03:23.616437 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.598693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.126:45811 (local address 127.0.0.1:38670)
0929 16:03:23.599447 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.599483 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:23.599675 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:23.600065 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:23.600072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:23.600948 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:23.601288 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.601301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.602540 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.602544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.604114 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.604134 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.604980 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.604988 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.605218 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.605888 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:23.605929 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:23.608510 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:23.611993 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.612002 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.612011 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.612637 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.613112 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.613117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.613120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.613330 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.613964 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:23.613971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:23.614241 (+ 270us) client_negotiation.cc:769] Sending connection context
0929 16:03:23.614581 (+ 340us) client_negotiation.cc:240] Negotiation successful
0929 16:03:23.615271 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":146,"thread_start_us":340,"threads_started":1}
I20260929 16:03:23.618124 15177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697803, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.126@KRBTEST.COM
I20260929 16:03:23.632532 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.618948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.125:40069 (local address 127.0.0.1:39982)
0929 16:03:23.619406 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.619422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:23.619557 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:23.619957 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:23.619961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:23.620402 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:23.620542 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.620551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.621819 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.621825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.622973 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.622985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.624006 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.624024 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.624232 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.624852 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:23.624876 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:23.626778 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:23.629947 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.629952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.629956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.630509 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.630913 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.630919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.630922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.631110 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.631679 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:23.631685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:23.631883 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 16:03:23.632044 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 16:03:23.632288 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358}
I20260929 16:03:23.633008 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.618938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.23.222.125:40069)
0929 16:03:23.619321 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.619329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.619524 (+ 195us) server_negotiation.cc:407] Connection header received
0929 16:03:23.619732 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.619739 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.619825 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.619924 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.620697 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.621676 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.623127 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.623848 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.623986 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.626959 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.626972 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.626976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.627007 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.629807 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.630641 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.630645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.630648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.630793 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.631219 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.631222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.631224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.631542 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.631669 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.632171 (+ 502us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.632414 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260929 16:03:23.634055 15177 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
I20260929 16:03:23.635005 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.619564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.23.222.126:45811)
0929 16:03:23.619734 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.619744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.619757 (+ 13us) server_negotiation.cc:407] Connection header received
0929 16:03:23.619926 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.619931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.620001 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.620101 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.621426 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.622435 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.623982 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.624585 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.624722 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.628784 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.628790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.628793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.628824 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.631919 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.632833 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.632839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.632842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.633000 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.633481 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.633485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.633488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.633833 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.634194 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.634472 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.634704 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697803, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.124@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.222.125@KRBTEST.COM
I20260929 16:03:23.647691 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.634790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.23.222.124:36451)
0929 16:03:23.635032 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.635091 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.635110 (+ 19us) server_negotiation.cc:407] Connection header received
0929 16:03:23.635299 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.635307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.635385 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.635473 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.636215 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.637198 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.638424 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.638975 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.639075 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.642324 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.642331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.642343 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.642372 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.645009 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.645635 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.645643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.645647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.645758 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.646113 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.646119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.646122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.646520 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.646673 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.647046 (+ 373us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.647326 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260929 16:03:23.647886 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.634734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50726)
0929 16:03:23.634903 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.634923 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:23.635033 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:23.635530 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:23.635535 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:23.635901 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:23.636069 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.636084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.637320 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.637325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.638278 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.638290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.639131 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.639137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.639381 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.639980 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:23.639997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:23.642114 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:23.645140 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.645148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.645152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.645520 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.645865 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.645872 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.645876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.646009 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.646645 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:23.646649 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:23.646798 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 16:03:23.646948 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 16:03:23.647478 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 16:03:23.651603 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.637073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.23.222.125:40069)
0929 16:03:23.637276 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.637281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.637297 (+ 16us) server_negotiation.cc:407] Connection header received
0929 16:03:23.637342 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.637346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.637419 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.637512 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.638773 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.639756 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.640980 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.641412 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.641560 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.645715 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.645721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.645724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.645756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.649012 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.649662 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.649669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.649672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.649809 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.650202 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.650208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.650211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.650473 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.650754 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.651025 (+ 271us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.651276 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 16:03:23.652885 15177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc: Bootstrap starting.
I20260929 16:03:23.656244 24441 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 16:03:23.656635 15177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:23.657661 15177 log.cc:824] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:23.660456 15177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc: No bootstrap required, opened a new log
I20260929 16:03:23.658725 15002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "876725c138094625b20077169557e52e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38676
I20260929 16:03:23.667567 15177 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } }
I20260929 16:03:23.667943 15177 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc79b753764943c79d606a9ceeae06bc, State: Initialized, Role: FOLLOWER
I20260929 16:03:23.668390 15177 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } }
I20260929 16:03:23.669405 15183 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.669811 15183 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:23.670485 15177 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [sys.catalog]: configured and running, proceeding with master startup.
W20260929 16:03:23.680270 15195 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:03:23.680409 15195 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:03:23.678097 15110 master_runner.cc:186] Error getting master registration for 127.23.222.126:45811: OK, instance_id { permanent_uuid: "876725c138094625b20077169557e52e" instance_seqno: 1790697802551795 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:03:23.686112 15110 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:03:23.694319 15028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } attempt: 4
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697802, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.126@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
I20260929 16:03:23.710505 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.695225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50742)
0929 16:03:23.695616 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.695639 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:23.695779 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:23.696261 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:23.696267 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:23.696834 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:23.697075 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.697089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.698555 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.698561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.699660 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.699672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.700737 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.700743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.700948 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.701588 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:23.701605 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:23.703872 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:23.707933 (+ 4061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.707939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.707943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.708318 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.708708 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.708713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.708717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.708876 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.709498 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:23.709503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:23.709651 (+ 148us) client_negotiation.cc:769] Sending connection context
0929 16:03:23.709798 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 16:03:23.710103 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260929 16:03:23.710520 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.695343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.23.222.124:36451)
0929 16:03:23.695532 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.695536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.695701 (+ 165us) server_negotiation.cc:407] Connection header received
0929 16:03:23.695969 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.695974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.696071 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.696211 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.697234 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.698400 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.699809 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.700589 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.700709 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.704077 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.704083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.704090 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.704136 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.707796 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.708446 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.708452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.708455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.708582 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.708991 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.708996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.708999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.709364 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.709532 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.709927 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.710139 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 16:03:23.716718 15028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e: Bootstrap starting.
I20260929 16:03:23.720813 15028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:23.722143 15028 log.cc:824] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:23.726449 15028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e: No bootstrap required, opened a new log
I20260929 16:03:23.734459 15028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } }
I20260929 16:03:23.734916 15028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 876725c138094625b20077169557e52e, State: Initialized, Role: FOLLOWER
I20260929 16:03:23.735519 15028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } }
I20260929 16:03:23.736790 15198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.737429 15198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:23.738135 15028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:23.738188 15002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 16:03:23.738366 15002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:23.742405 15002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } }
I20260929 16:03:23.747118 15152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "876725c138094625b20077169557e52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dc79b753764943c79d606a9ceeae06bc"
I20260929 16:03:23.748979 15152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:23.749469 15002 leader_election.cc:288] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [CANDIDATE]: Term 1 election: Requested vote from peers 18a484f6ac6f41edb3fdad7488dd855f (127.23.222.125:40069), dc79b753764943c79d606a9ceeae06bc (127.23.222.124:36451)
I20260929 16:03:23.752840 15152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 876725c138094625b20077169557e52e in term 1.
I20260929 16:03:23.753681 14976 leader_election.cc:302] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 876725c138094625b20077169557e52e, dc79b753764943c79d606a9ceeae06bc; no voters:
I20260929 16:03:23.754364 15198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 16:03:23.758428 15198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [term 1 LEADER]: Becoming Leader. State: Replica: 876725c138094625b20077169557e52e, State: Running, Role: LEADER
I20260929 16:03:23.759215 15198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } }
W20260929 16:03:23.762935 15209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 16:03:23.763731 15209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 16:03:23.767450 15212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 876725c138094625b20077169557e52e. Latest consensus state: current_term: 1 leader_uuid: "876725c138094625b20077169557e52e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.767968 15212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:23.770952 15215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 16:03:23.773924 15215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 16:03:23.750232 14960 master_runner.cc:186] Error getting master registration for 127.23.222.125:40069: OK, instance_id { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" instance_seqno: 1790697803058692 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 16:03:23.778293 14960 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 16:03:23.788838 15103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } attempt: 5
W20260929 16:03:23.791169 14981 tablet.cc:2369] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:03:23.794049 15152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [term 1 FOLLOWER]: Refusing update from remote peer 876725c138094625b20077169557e52e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:23.795733 15212 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 29 16:03:23 dist-test-slave-cq88 krb5kdc[14947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790697803, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.222.125@KRBTEST.COM for kudu/127.23.222.124@KRBTEST.COM
W20260929 16:03:23.815421 15131 tablet.cc:2369] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 16:03:23.821707 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.789578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.222.124:36451 (local address 127.0.0.1:50748)
0929 16:03:23.790043 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 16:03:23.790067 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 16:03:23.790210 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 16:03:23.791684 (+ 1474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 16:03:23.791688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 16:03:23.792207 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0929 16:03:23.792434 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.792447 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.794060 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.794067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.795463 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 16:03:23.795480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.800735 (+ 5255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.800742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 16:03:23.800991 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.802373 (+ 1382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 16:03:23.802400 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 16:03:23.806728 (+ 4328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 16:03:23.814463 (+ 7735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.814469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.814474 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.815708 (+ 1234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.817335 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 16:03:23.817342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 16:03:23.817347 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 16:03:23.817708 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 16:03:23.820023 (+ 2315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 16:03:23.820029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 16:03:23.820300 (+ 271us) client_negotiation.cc:769] Sending connection context
0929 16:03:23.820529 (+ 229us) client_negotiation.cc:240] Negotiation successful
0929 16:03:23.821040 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302}
I20260929 16:03:23.825918 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.790452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50748 (local address 127.23.222.124:36451)
0929 16:03:23.791162 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.791169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.791191 (+ 22us) server_negotiation.cc:407] Connection header received
0929 16:03:23.791273 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.791281 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.791361 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.791467 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.792768 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.793881 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.799532 (+ 5651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.800233 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.800373 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.806980 (+ 6607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.806987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.806992 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.807040 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.814110 (+ 7070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.816083 (+ 1973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.816089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.816093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.816903 (+ 810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.818046 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.818051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.818054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.819646 (+ 1592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.822810 (+ 3164us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.823424 (+ 614us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.824475 (+ 1051us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578}
I20260929 16:03:23.832846 15103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f: Bootstrap starting.
I20260929 16:03:23.839134 15103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f: Neither blocks nor log segments found. Creating new log.
I20260929 16:03:23.844553 15103 log.cc:824] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f: Log is configured to *not* fsync() on all Append() calls
I20260929 16:03:23.855940 15183 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 876725c138094625b20077169557e52e. Latest consensus state: current_term: 1 leader_uuid: "876725c138094625b20077169557e52e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.856340 15183 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:23.869853 15103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f: No bootstrap required, opened a new log
I20260929 16:03:23.874369 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.791698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.23.222.125:40069)
0929 16:03:23.792679 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.792685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.847791 (+ 55106us) server_negotiation.cc:407] Connection header received
0929 16:03:23.848088 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.848094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.848179 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.848253 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.852555 (+ 4302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.853716 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.861404 (+ 7688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.862145 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.862263 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.866342 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.866351 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.866355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.866398 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.870514 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.871287 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.871293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.871297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.871465 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.871922 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.871927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.871931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.872383 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.872694 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.873679 (+ 985us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.873983 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875}
I20260929 16:03:23.879805 15183 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "876725c138094625b20077169557e52e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.880079 15183 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:23.885085 15221 mvcc.cc:205] Tried to move back new op lower bound from 7334698204318523392 to 7334698204204814336. Current Snapshot: MvccSnapshot[applied={T|T < 7334698204318523392 or (T in {7334698204318523392})}]
I20260929 16:03:23.885936 15103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } }
I20260929 16:03:23.886680 15103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a484f6ac6f41edb3fdad7488dd855f, State: Initialized, Role: FOLLOWER
I20260929 16:03:23.887305 15103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } }
I20260929 16:03:23.889102 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.846750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.23.222.124:36451)
0929 16:03:23.847357 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.847364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.860641 (+ 13277us) server_negotiation.cc:407] Connection header received
0929 16:03:23.860710 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.860717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.860794 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.860878 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.863309 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.864353 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.866982 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.867744 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.867877 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.877324 (+ 9447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.877331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.877336 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.877378 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.882018 (+ 4640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.884534 (+ 2516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.884541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.884544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.884695 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.886347 (+ 1652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.886351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.886356 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.886747 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.886988 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.888196 (+ 1208us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.888548 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
I20260929 16:03:23.896665 15227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.899050 15077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "876725c138094625b20077169557e52e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "18a484f6ac6f41edb3fdad7488dd855f"
I20260929 16:03:23.899533 15076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [term 0 FOLLOWER]: Advancing to term 1
I20260929 16:03:23.899046 15103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 16:03:23.899919 15198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "876725c138094625b20077169557e52e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.900249 15198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:23.901301 15227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:23.902084 15215 catalog_manager.cc:1347] Generated new cluster ID: cd8fcf42063e4481b3272b79f18956b6
I20260929 16:03:23.902186 15215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 16:03:23.903846 15076 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [term 1 FOLLOWER]: Refusing update from remote peer 876725c138094625b20077169557e52e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 16:03:23.904861 15212 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 16:03:23.906482 15030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:23.792803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38684 (local address 127.23.222.126:45811)
0929 16:03:23.793119 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:23.793125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:23.850779 (+ 57654us) server_negotiation.cc:407] Connection header received
0929 16:03:23.852268 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:23.852277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:23.852380 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:23.852474 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:23.863755 (+ 11281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.865009 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.870145 (+ 5136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:23.885477 (+ 15332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:23.885637 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:23.889859 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 16:03:23.889867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 16:03:23.889872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 16:03:23.889914 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 16:03:23.894336 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.895213 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.895279 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.895287 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.895783 (+ 496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 16:03:23.896982 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 16:03:23.896992 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 16:03:23.896997 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 16:03:23.897819 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 16:03:23.905636 (+ 7817us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:23.905799 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 16:03:23.906099 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260929 16:03:23.907214 15077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 876725c138094625b20077169557e52e in term 1.
I20260929 16:03:23.923030 15215 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 16:03:23.925292 15215 catalog_manager.cc:1504] Loading token signing keys...
I20260929 16:03:23.940312 15227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 876725c138094625b20077169557e52e. Latest consensus state: current_term: 1 leader_uuid: "876725c138094625b20077169557e52e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.940717 15227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:23.941319 15232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "876725c138094625b20077169557e52e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.943058 15198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "876725c138094625b20077169557e52e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "876725c138094625b20077169557e52e" member_type: VOTER last_known_addr { host: "127.23.222.126" port: 45811 } } peers { permanent_uuid: "18a484f6ac6f41edb3fdad7488dd855f" member_type: VOTER last_known_addr { host: "127.23.222.125" port: 40069 } } peers { permanent_uuid: "dc79b753764943c79d606a9ceeae06bc" member_type: VOTER last_known_addr { host: "127.23.222.124" port: 36451 } } }
I20260929 16:03:23.943403 15198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e [sys.catalog]: This master's current role is: LEADER
I20260929 16:03:23.948035 15215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 876725c138094625b20077169557e52e: Generated new TSK 0
I20260929 16:03:23.949292 15215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 16:03:23.950711 15232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 16:03:24.014650 15245 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 16:03:24.046993 15030 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.038219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.23.222.126:45811)
0929 16:03:24.038440 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.038449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.044324 (+ 5875us) server_negotiation.cc:407] Connection header received
0929 16:03:24.044514 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.044524 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.044687 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.044872 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.046590 (+ 1718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38694: BlockingRecv error: recv got EOF from 127.0.0.1:38694 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 16:03:24.050498 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.038064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.23.222.125:40069)
0929 16:03:24.038298 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.038307 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.048145 (+ 9838us) server_negotiation.cc:407] Connection header received
0929 16:03:24.048488 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.048500 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.048637 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.048740 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.050163 (+ 1423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40012: BlockingRecv error: recv got EOF from 127.0.0.1:40012 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 16:03:24.053898 15180 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.051984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.23.222.124:36451)
0929 16:03:24.052173 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.052180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.052351 (+ 171us) server_negotiation.cc:407] Connection header received
0929 16:03:24.052568 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.052573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.052640 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.052755 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.053631 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50770: BlockingRecv error: recv got EOF from 127.0.0.1:50770 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 16:03:24.066285 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.063484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.23.222.125:40069)
0929 16:03:24.063739 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.063749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.064249 (+ 500us) server_negotiation.cc:407] Connection header received
0929 16:03:24.064615 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.064622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.064749 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.064845 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.065953 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40024: BlockingRecv error: recv got EOF from 127.0.0.1:40024 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 16:03:24.101423 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.095914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.23.222.125:40069)
0929 16:03:24.096116 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.096126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.099770 (+ 3644us) server_negotiation.cc:407] Connection header received
0929 16:03:24.100004 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.100010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.100105 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.100215 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.101210 (+ 995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40032: BlockingRecv error: recv got EOF from 127.0.0.1:40032 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 16:03:24.109241 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.106763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.23.222.125:40069)
0929 16:03:24.106948 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.106955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.107497 (+ 542us) server_negotiation.cc:407] Connection header received
0929 16:03:24.107805 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.107811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.107906 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.107984 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.109000 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40042: BlockingRecv error: recv got EOF from 127.0.0.1:40042 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 16:03:24.121269 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.117612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.23.222.125:40069)
0929 16:03:24.117815 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.117821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.119791 (+ 1970us) server_negotiation.cc:407] Connection header received
0929 16:03:24.119979 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.119986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.120099 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.120225 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.121017 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40054: BlockingRecv error: recv got EOF from 127.0.0.1:40054 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260929 16:03:24.147338 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.144894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.23.222.125:40069)
0929 16:03:24.145107 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.145114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.145691 (+ 577us) server_negotiation.cc:407] Connection header received
0929 16:03:24.145887 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.145894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.145999 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.146127 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.147032 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40070: BlockingRecv error: recv got EOF from 127.0.0.1:40070 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 16:03:24.169438 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.166922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.23.222.125:40069)
0929 16:03:24.167118 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.167126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.167665 (+ 539us) server_negotiation.cc:407] Connection header received
0929 16:03:24.167880 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.167888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.168003 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.168167 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.169128 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40080: BlockingRecv error: recv got EOF from 127.0.0.1:40080 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 16:03:24.207810 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.205344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.23.222.125:40069)
0929 16:03:24.205542 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.205551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.206151 (+ 600us) server_negotiation.cc:407] Connection header received
0929 16:03:24.206375 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.206390 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.206509 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.206633 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.207543 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40086: BlockingRecv error: recv got EOF from 127.0.0.1:40086 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 16:03:24.277904 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.275896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.23.222.125:40069)
0929 16:03:24.276114 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.276121 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.276537 (+ 416us) server_negotiation.cc:407] Connection header received
0929 16:03:24.276726 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.276733 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.276837 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.276969 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.277689 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40102: BlockingRecv error: recv got EOF from 127.0.0.1:40102 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 16:03:24.412175 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.410190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.23.222.125:40069)
0929 16:03:24.410453 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.410462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.410725 (+ 263us) server_negotiation.cc:407] Connection header received
0929 16:03:24.410970 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.410974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.411049 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.411169 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.411957 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40106: BlockingRecv error: recv got EOF from 127.0.0.1:40106 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260929 16:03:24.674598 15107 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 16:03:24.672517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40116 (local address 127.23.222.125:40069)
0929 16:03:24.672788 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:24.672795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:24.673038 (+ 243us) server_negotiation.cc:407] Connection header received
0929 16:03:24.673213 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:24.673217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:24.673300 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:24.673478 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 16:03:24.674387 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40116: BlockingRecv error: recv got EOF from 127.0.0.1:40116 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 16:03:24.682505 15195 catalog_manager.cc:1259] Loaded cluster ID: cd8fcf42063e4481b3272b79f18956b6
I20260929 16:03:24.682621 15195 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc: loading cluster ID for follower catalog manager: success
I20260929 16:03:24.685160 15195 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc: acquiring CA information for follower catalog manager: success
I20260929 16:03:24.687181 15195 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dc79b753764943c79d606a9ceeae06bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:03:25.019400 15245 catalog_manager.cc:1259] Loaded cluster ID: cd8fcf42063e4481b3272b79f18956b6
I20260929 16:03:25.019520 15245 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f: loading cluster ID for follower catalog manager: success
I20260929 16:03:25.022214 15245 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f: acquiring CA information for follower catalog manager: success
I20260929 16:03:25.024197 15245 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18a484f6ac6f41edb3fdad7488dd855f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 16:03:25.210409 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:25.196897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.23.222.125:40069)
0929 16:03:25.204175 (+ 7278us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:25.204180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:25.204201 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:03:25.204265 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:25.204269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:25.204413 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:25.204533 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:25.205088 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.205901 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.207931 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.208403 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.208502 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:25.209120 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:25.209672 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:25.209775 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:25.209926 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 16:03:25.209995 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7177,"thread_start_us":7012,"threads_started":1}
I20260929 16:03:25.229251 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:25.222950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.23.222.124:36451)
0929 16:03:25.223712 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:25.223717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:25.223740 (+ 23us) server_negotiation.cc:407] Connection header received
0929 16:03:25.223854 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:25.223859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:25.224027 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:25.224205 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:25.224654 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.225402 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.226880 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.227413 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.227548 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:25.228113 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:25.228637 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:25.228794 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:25.228854 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 16:03:25.228936 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":527,"threads_started":1}
I20260929 16:03:25.247570 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:25.236110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.23.222.126:45811)
0929 16:03:25.239671 (+ 3561us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:25.239710 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:25.239731 (+ 21us) server_negotiation.cc:407] Connection header received
0929 16:03:25.239837 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:25.239840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:25.239991 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:25.240166 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:25.240789 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.241954 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.244272 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.244889 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.245259 (+ 370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:25.245595 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:25.246050 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:25.247056 (+ 1006us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:25.247113 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 16:03:25.247202 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3465,"thread_start_us":3329,"threads_started":1}
I20260929 16:03:25.248075 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:25.235657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.23.222.125:40069)
0929 16:03:25.235917 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:25.235926 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:25.240951 (+ 5025us) server_negotiation.cc:407] Connection header received
0929 16:03:25.241207 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:25.241213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:25.241323 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:25.241385 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:25.242188 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.243380 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.245773 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.246417 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.246521 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:25.247200 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:25.247338 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:25.247402 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:25.247672 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 16:03:25.247755 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 16:03:25.253073 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 16:03:25.241327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.23.222.124:36451)
0929 16:03:25.241508 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 16:03:25.241515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 16:03:25.246860 (+ 5345us) server_negotiation.cc:407] Connection header received
0929 16:03:25.246992 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 16:03:25.246998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 16:03:25.247102 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 16:03:25.247200 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 16:03:25.248117 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.249234 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.251182 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 16:03:25.251806 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 16:03:25.251940 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 16:03:25.252325 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 16:03:25.252475 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 16:03:25.252587 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 16:03:25.252725 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 16:03:25.252805 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 16:03:25.257848 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 14960
I20260929 16:03:25.258986 15022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:25.369359 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 14960
I20260929 16:03:25.391216 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 15035
I20260929 16:03:25.392416 15097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:25.502502 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 15035
I20260929 16:03:25.523211 24441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskGayxzf/build/asan/bin/kudu pid 15110
I20260929 16:03:25.524547 15172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 16:03:25.631518 24441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGayxzf/build/asan/bin/kudu with pid 15110
2026-09-29T16:03:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4926 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9461 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (373882 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST